Visual saliency estimation by nonlinearly integrating features using region covariances

Journal of Vision, 13(4):11, 1-20, 2013

Erkut Erdem     Aykut Erdem

The proposed saliency model. The input image is first decomposed into non-overlapping regions, and then the saliency of each region is measured by examining its surrounding regions. The salient regions are those that are highly dissimilar to their neighboring regions in terms of their covariance representations based on color, orientation, and spatial features. In the saliency map computed by the proposed model, the fish pops out from the complex background, cf. Itti's saliency map (Itti, Koch, & Niebur, 1998).

Abstract

To detect visually salient elements of complex natural scenes, computational bottom-up saliency models commonly examine several feature channels such as color and orientation in parallel. They compute a separate feature map for each channel and then linearly combine these maps to produce a master saliency map. However, only a few studies have investigated how different feature dimensions contribute to the overall visual saliency. We address this integration issue and propose to use covariance matrices of simple image features (known as region covariance descriptors in the computer vision community; Tuzel, Porikli, and Meer, 2006) as meta-features for saliency estimation. As low- dimensional representations of image patches, region covariances capture local image structures better than standard linear filters, but more importantly, they naturally provide nonlinear integration of different features by modeling their correlations. We also show that first-order statistics of features could be easily incorporated to the proposed approach to improve the performance. Our experimental evaluation on several benchmark data sets demonstrate that the proposed approach outperforms the state-of-art models on various tasks including prediction of human eye fixations, salient object detection, and image-retargeting.

Paper

E. Erdem and A. Erdem. Visual saliency estimation by nonlinearly integrating features using region covariances. Journal of Vision, 13(4):11, 1-20, 2013.

Code

Code for predicting the saliency maps.

Supplementary Material

Saliency Maps for the candles image
Predicting human eye fixations
Detecting salient objects
Image retargeting by seam-carving
Psychological Patterns

Bibtex

@article{Erdem:JoV2013,
author = {Erkut Erdem and Aykut Erdem},
title = {Visual saliency estimation by nonlinearly integrating features using region covariances},
journal = {Journal of Vision},
year = {2013},
volume = {13},
number = {4},
pages = {1--20}
}

Related papers

B. Celikkale, A. Erdem and E. Erdem. Visual Attention-driven Spatial Pooling for Image Memorability. IEEE Computer Vision and Pattern Recognition Workshops (CVPRW), Portland, Oregon, USA, June 2013.

Acknowledgments

This research was supported in part by The Scientific and Technological Research Council of Turkey (TUBITAK), Career Development Award 112E146.