Course Project

An integral part of the course is the class project (25% of the grade), which gives students a chance to apply the algorithms discussed in class to a research oriented project.

Theme of this year's projects will be image colorization.

Image colorization aims to convert a given gray image to a plausible color version. Since a gray pixel might be mapped to more than one color (one t-shirt may be red, black or blue), many works have been attacked to this problem usually by utilizing a reference either with a human intervention or a color image. In the most recent work [1], they introduce an end-to-end deep network which colorizes image in an exemplar-based manner (by referencing a color image). Please refer to [1] for further information about image colorization and related works. Moreover, you can find some selected approaches below which are given in [1] as well.

Note: You may utilize the codes of the works (in Python or MATLAB) but you should implement your idea over the given methods.

1. Scribble-based colorization

In this type of image colorization, proposed studies use human assistance. For example, in [2], they utilize color scribbles which are annotated by the artists on gray images. After that, their method propagates through pixels and colorizes the image. The idea behind this approach is: neighboring pixels that have similar intensities should have the same color. You can investigate the more recent works over this pioneering study.

2. Example-based colorization

Exempler-based methods utilize a reference image which is very similar to given gray image in terms of semantic or visual structures. The earliest work [3] considers global color statistics but not the spatial information of the pixel values. Hence, it may give poor colorization results. The more recent work [4] uses deep features of the images to match the semantic structures and colorizes accordingly from the reference image. The previously mentioned work [1] benefits from this approach in an end-to-end manner. You can explore the other similar works over mentioned studies.

3. Learning-based colorization

Some studies focus on learning the colorization process in a classification/regression manner. They basically train their models to learn the mapping between a grayscale input image and color distribution. The recent work [5] proposes a Convolutional Neural Network (CNN) for this purpose. They basically train their network with a large-scale image data and then gives testing results over this model.

References

[1] He, M., Chen, D., Liao, J., Sander, P.V. and Yuan, L., 2018. Deep exemplar-based colorization. ACM Transactions on Graphics (TOG)
[2] Levin, A., Lischinski, D. and Weiss, Y., 2004. Colorization using optimization. ACM Transactions on Graphics (TOG)
[3] Welsh, T., Ashikhmin, M. and Mueller, K., 2002, July. Transferring color to greyscale images. ACM Transactions on Graphics (TOG)
[4] He, M., Liao, J., Yuan, L. and Sander, P.V., 2017. Neural color transfer between images. arXiv preprint arXiv:1710.00756.
[5] Zhang, R., Isola, P. and Efros, A.A., 2016. Colorful image colorization. European Conference on Computer Vision

Deliverables

• Proposals: November 13, 2018.
• Project progress reports: December 18, 2018
• Final project presentations: January 8, 2019
• Final reports: January 15, 2019
  
  

In preparing your progress and final project reports, you should use the provided LaTeX template and submit them electronically in PDF format. Late submissions will be penalized.

Collaboration Policy

Each project should be done in groups of 2-3 students. Of course, there may be some exceptions, depending on the enrollment. Note that students without a team will be randomly assigned to one project group.

Grading

• Proposal (2%)
• Blog posts (3%)
• GitHub commits and meetings with the TA (3%)
• Progress report (4%)
• Presentation (6%)
• Final report and code (7%)

Project Proposal

Each project group should submit a half page project proposal on their specific project idea by November 13, 2018. The proposal should provide

• A brief overview of the proposed image colorization approach,
• A list of related papers.

Blog posts/GitHub commits/Meetings with the TA

Each project group should maintain a blog sharing their steady progress, ideas, and experiments, and must write at least one blog post per week (excluding the exam week). Moreover, they will regularly meet with the TA to discuss their progress and get feedback. Each group should maintain a GitHub repository for their project (must be viewable to the TA and the instructor). The frequency of your commits to GitHub will also be graded.

Progress Report

Due: December 18, 2018 (11:59pm)

Each student should submit a project progress report by December 18, 2018. The report should be 3-4 pages and should describe the following points as clearly as possible:

• Problem to be addressed. Give a short description of the problem that you will explore. Explain why you find it interesting.
• Related work. Briefly review the major works related to your research topic.
• Methodology to be employed. Describe the image colorization method that is expected to form the basis of the project. State whether you will extend an existing method or you are going to devise your own approach.
• Experimental evaluation. Briefly explain how you will evaluate your results. State which dataset(s) you will employ in your evaluation. Provide your preliminary results (if any).

Project Presentations

Due: January 8, 2019 (in class)

Each project group will have ~10 mins to present their work in class. The suggested outline for the presentations are as follows:

• High-level overview of the paper (main contributions)
• Problem statement and motivation (clear definition of the problem, why it is interesting and important)
• Key technical ideas (overview of the approach)
• Experimental set-up (datasets, evaluation metrics, applications)
• Strengths and weaknesses (discussion of the results obtained)

In addition to classroom presentations, each group should also prepare an engaging video presentation of their work using online tools such as PowToon, moovly or GoAnimate. The deadline is January 18, 2019.

Final Report

Due: January 15, 2019 (11:59pm)

As the last deliverable of the course project, each group is expected to submit a project report prepared using the style files provided in the course web page. The report should be 6-8 pages and should be structured as a research paper. It will be graded based on clarity of presentation and technical content. A typical organization of a report might follow:

• Title, Author(s).
• Abstract.
• Introduction. This section introduces the problem that you investigated by providing a general motivation and briefly discusses the approach(es) that you explored to solve this problem.
• Related Work. This section discusses relevant literature for your project topic.
• The Approach. This section gives the technical details about your project work. You should describe the representation(s) and the algorithm(s) that you employed or proposed as detailed and specific as possible.
• Experimental Results. This section presents some experiments in which you analyze the performance of the approach(es) you proposed or explored. You should provide a qualitative and/or quantitative analysis, and comment on your findings. You may also demonstrate the limitations of the approach(es).
• Conclusions. This section summarizes all your project work, focusing on the key results you obtained. You may also suggest possible directions for future work.
• References. This section gives a list of all related work you reviewed or used.