Vicente Ordonez will work on a 2007 paper about detecting salient objects in images. The paper uses multiscale contrast, center surround histograms, and color spatial distribution as visual attention features. These features are combined using conditional random fields trained on a labeled dataset to determine salient regions. Ordonez implements the features and achieves results similar to the original paper, with computation times of several seconds per image. The center surround histogram feature gives the highest precision for detecting salient objects.

1. Visual Attention: Detecting Saliency on ImagesVicente OrdonezDepartment of Computer ScienceState University of New YorkStony Brook, NY 11790

2. I will be working mainly on the following paperLearning to Detect a Salient Object. T. Liu, J. Sun, N. Zheng, X. Tang, H. Shum. (Xian Jiaotong University and Microsoft Research Asia) from CVPR 2007. http://research.microsoft.com/en-us/um/people/jiansun/papers/SalientDetection_CVPR07.pdf

3. What is Saliency? What is Visual Attention?“Everyone knows what attention is...”—William James, 1890

4. This is a problem of…Arbitrary object detection?Background / Foreground segmentation?Modeling Visual Attention?

5. The MethodFeatures: Multiscale Contrast (Done!)Center surround histogram (Mostly Done!) (Done!)Color spatial distribution (Done!)Supervised learning using Conditional Random Fields to determine the parameters to combine the features obtained above. (Done!) [I will use a labeled dataset of 5000 images provided by Microsoft Research Asia!]

6. Multiscale Contrast FunctionGenerate the Gaussian Pyramid for the input image.For each level in the pyramid Do gaussian blurringDo resamplingI’m using a 6 levels Gaussian pyramid for each RGB channel.

7. How a Gaussian pyramid looks likeFigure from David Forsyth

8. Generate contrast maps for each level of the Pyramid.Sum all of the results to produce the final multiscale contrast map.The two steps mentioned above are described in this formula:Multiscale Contrast Function

9. Input image

10. Contrast maps

11. Contrast mapsOriginal imageContrast map at level 1Contrast map at level 4Contrast map at level 6

12. Multiscale Contrast Map Output

13. Center Surround Histogram FeatureFor each pixel in the image

14. For each possible rectangle with a reasonable size and aspect ratio

15. Create a surrounding rectangle and calculate the histogram of the rectangle and the surrounding area.

16. Pick and record the rectangle that maximizes the Chi-Square distance between the two histograms calculated above and also record the Chi-Square distance.Center Surround Histogram Feature

17. Center Surround Histogram FeatureThe algorithm as described before is computationally expensive… It is required to use a technique called Integral Histogram. It allows you fast calculation of the histogram of any given rectangular region of an image.The algorithm was introduced in:“Integral Histogram: A Fast Way to Extract Histograms in Cartesian Spaces” by FatihPorikli, Mitsubishi Electric Research Lab in CVPR 2005.

18. Center Surround Histogram FeatureUse the Chi Square Distances Map and the Map of Most Salient Rectangle Regions per pixel to generate the Center Surround Histogram Feature using the next formula:

19. Center Surround HistogramResults Using my Implementation (15.2 sec, size = 245x384)Results Reported in the Paper

20. Center Surround HistogramResults Using my Implementation (13.6 sec, size = 247x346)Results Reported in the Paper

21. Center Surround HistogramResults Using my Implementation (10.2 sec, size = 248x277)

22. More Results

23. More Results

24. More results

25. More Results

26. More Results

27. More Results

28. More Results

29. More Results

30. More Results

31. More Results

32. More Results

33. Color Spatial Distribution

34. Color Spatial DistributionMake an initial clustering of the colors in the image using k-means. Further refine the clusters by using Gaussian Mixture Models. The Gaussian Mixture Model parameters are calculated using the EM algorithm.I am using 5 clusters (5 colors) per image. And the results look similar to those presented in the paper with an execution time of around 17 seconds per image.

35. Color Spatial DistributionCalculate the vertical variance of the horizontal positions of the pixels for each cluster. And then the same for the vertical positions. Sum the variances and use this value to weight more those clusters with less spatial variance.Penalize the clusters that contain the majority of its pixels away from the center of the image.

36. Color Spatial Distribution

37. Color Spatial Distribution

38. Color Spatial Distribution

39. Color Spatial Distribution

40. Color Spatial Distribution

41. Color Spatial Distribution

42. Color Spatial Distribution

43. Color Spatial Distribution

44. Combine Features Together

45. Conditional Random Field Training and InferenceAccelerated Training of Conditional Random Fields with Stochastic Meta-Descent S Vishwanathan, N. Schraudolph, M. Schmidt, K. Murphy. ICML'06 (Intl Conf on Machine Learning). I did the training using this toolbox from the above paper:http://people.cs.ubc.ca/~murphyk/Software/CRF/crf.html

46. Mask outputs using CRF inferenceInput M-Contrast-map Center Surr. Hist. Color Spatial Var.Input Combined features Ground truth

47. Mask outputs using CRF inferenceInput M-Contrast-map Center Surr. Hist. Color Spatial Var.Input Combined features Ground truth

48. Mask outputs using CRF inferenceInput M-Contrast-map Center Surr. Hist. Color Spatial Var.Input Combined features Ground truth

49. Mask outputs using CRF inferenceInput M-Contrast-map Center Surr. Hist. Color Spatial Var.Input Combined features Ground truth