Course Project

An integral part of the course is the class project (35% of the grade), which gives students a chance to apply the algorithms discussed in class to a research oriented project. The theme of the projects will be announced soon! This semester the theme is Multimodal Machine Learning.

Some example project titles are as follows:

• Multimodal Summarization of User Videos. Generating video snippets of a given video using video frames, audio and transcripts.
• Image captioning. Generating natural language descriptions of a given image.
• Multimodal Emotion Recognition. Recognizing human emotional states from speech and visual behaviors.

Some survey papers on this years project theme

1. Dhanesh Ramachandram, Graham W. Taylor, Louis-Philippe Morency. Deep multimodal learning: A survey on recent advances and trends. IEEE Signal Processing Magazine, Volume 34, Issue 6, pp 96-108, 2017.
2. Yingming Li, Ming Yang, Zhongfei Zhang. A survey of multi-view representation learning. IEEE Transactions on Knowledge and Data Engineering, Volume 31, Issue 10, pp 1863–1883, 2018.
3. Tadas Baltrušaitis, Chaitanya Ahuja, Louis-Philippe Morency. Multimodal Machine Learning: A Survey and Taxonomy. IEEE Transactions on Pattern Analysis and Machine Intelligence, Volume 41, Issue 2, pp 423-443, February 2019.
4. Chao Zhang, Zichao Yang, Xiaodong He, Li Deng. Multimodal intelligence: Representation learning, information fusion, and applications. IEEE Journal of Selected Topics in Signal Processing, 2020.
5. Aditya Mogadala, Marimuthu Kalimuthu, Dietrich Klakow. Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods. Journal of Artificial Intelligence Research, Volume 71, pp 1183-1317, August 2021.
6. Letitia Parcalabescu, Nils Trost, Anette Frank. What is Multimodality?. Workshop on Multimodal Semantic Representations (MMSR) 2021.

Resources

• Tutorial on Multimodal Machine Learning by Louis-Philippe Morency and Tadas Baltrušaitis, ACL 2017
• NAACL2021 Workshop on Multimodal Artificial Intelligence (MAI-Workshop)
• CVPR 2021 Multimodal Learning and Applications Workshop

In your projects, you may use a dataset available on the web or collect your own data. However, if you choose the latter option, you must you must keep in mind that data collection can be fun and exciting, but it is also time-consuming.

Please also keep in mind that complexity of your project topic will be taken into consideration while grading.

Software and Libraries

You are encouraged to learn and use the following machine learning and deep learning frameworks in your projects. Links to some useful NLP tools are also provided.

• PyTorch: An Open Source Machine Learning Library
• Tensorflow: An Open Source Software Library for Machine Intelligence
• Caffe: A deep learning framework made with expression, speed, and modularity in mind
• Theano: A Python framework for fast computation of mathematical expressions.
• Keras: Keras: Deep Learning library for Theano and TensorFlow
• MatConvNet: CNNs for MATLAB
• mxnet: Flexible and Efficient Library for Deep Learning
• Torch: A scientific computing framework for LuaJIT
• LIBSVM: A Library for Support Vector Machines
• scikit-learn: Machine learning in Python
• Stanford CoreNLP: A suite of core NLP tools
• NLTK: Natural Language Toolkit

Deliverables

• Proposals: November 9, 2021.
• Project progress reports: December 16, 2021
• Final project presentations: December 30, 2021
• Final reports: January 2, 2022
  
  

In preparing your progress and final project reports, you should use the provided LaTeX template and submit them electronically in PDF format. Late submissions will be penalized.

Collaboration Policy

Each project should be done in pairs. Of course, there may be some exceptions, depending on the enrollment. Note that students without a team will be randomly assigned to one project group.

Grading

• Proposal (2%)
• Blog posts (4%)
• GitHub commits and meetings with TAs(5%)
• Progress report (6%)
• Presentation (8%)
• Final report and code (10%)

Project Proposal

Each project group should submit a half page project proposal on their specific project idea by November 9, 2021. The proposal should provide

• The research topic to be investigated,
• What data you will use,
• A list of related papers.

Blog posts/GitHub commits/Meetings with TAs

Each project group should maintain a blog sharing their steady progress, ideas, and experiments, and they must write at least one blog post per week (excluding exam weeks). Moreover, they will regularly meet with TAs to discuss their progress and get feedback. Each group should maintain a GitHub repository for their project (must be viewable to the TAs and instructor). The frequency of your commits to GitHub will also be graded.

Progress Report

Due: December 16, 2021 (11:59pm)

Each project group should submit a project progress report by December 16, 2021. The report should be 3-4 pages and should describe the following points as clearly as possible:

• Problem to be addressed. Give a short description of the problem that you will explore. Explain why you find it interesting.
• Related work. Briefly review the major works related to your research topic.
• Methodology to be employed. Describe the machine learning method that is expected to form the basis of the project. State whether you will extend an existing method or you are going to devise your own approach.
• Experimental evaluation. Briefly explain how you will evaluate your results. State which dataset(s) you will employ in your evaluation. Provide your preliminary results (if any).

Project Presentations

Due: December 30, 2021 (in class)

Each project group will have ~10bbm mins to present their work in class. The suggested outline for the presentations are as follows:

• High-level overview of the paper (main contributions)
• Problem statement and motivation (clear definition of the problem, why it is interesting and important)
• Key technical ideas (overview of the approach)
• Experimental set-up (datasets, evaluation metrics, applications)
• Strengths and weaknesses (discussion of the results obtained)

In addition to classroom presentations, each group should also prepare an engaging video presentation of their work using online tools such as PowToon, moovly or GoAnimate. The deadline is January 2, 2022.

Final Report

Due: January 2, 2022 (11:59pm)

As the last deliverable of the course project, each group is expected to submit a project report prepared using the style files provided in the course web page. The report should be 6-8 pages and should be structured as a research paper. It will be graded based on clarity of presentation and technical content. A typical organization of a report might follow:

• Title, Author(s).
• Abstract.
• Introduction. This section introduces the problem that you investigated by providing a general motivation and briefly discusses the approach(es) that you explored to solve this problem.
• Related Work. This section discusses relevant literature for your project topic.
• The Approach. This section gives the technical details about your project work. You should describe the representation(s) and the algorithm(s) that you employed or proposed as detailed and specific as possible.
• Experimental Results. This section presents some experiments in which you analyze the performance of the approach(es) you proposed or explored. You should provide a qualitative and/or quantitative analysis, and comment on your findings. You may also demonstrate the limitations of the approach(es).
• Conclusions. This section summarizes all your project work, focusing on the key results you obtained. You may also suggest possible directions for future work.
• References. This section gives a list of all related work you reviewed or used.