ICCV 2011  Scale and Object Aware ImageRetargeting for Thumbnail Browsing            Jin Sun, Haibin Ling  Temple Universi...
Introduction• Image browsing tasks• Tiny thumbnails: a fixed small size• Scaling      – bring difficulties in searching an...
Thumbnail Browsing• Thumbnail scales      – Studies have shown that scales can have        significant effects on human vi...
Scale and Object Aware Retargeting• Thumbnail scales      – An image perceived by human vision system      – A new scale d...
Scale and Object Aware Retargeting• A continuous retargeting algorithm but uses  discrete retargeting to guide the warping...
Overview : Task Formulation• Retargeting problem:                 J  F ( I , S ( I ))• Seam carving(SC)                 ...
Overview : Framework Overview•   Scale and object aware image retargeting(SOAP): F   so•   Warping function: thin-plate-sp...
Flow chart of the proposed method2012/6/19                               8
Scale-dependent Saliency•   Original image size•   Display size                  so        s D                          s...
Scale-dependent Saliency• How well such a system preserves the image  information?• We want to make the foreground object/...
Scale-dependent Saliency• the perceived image, denoted as I p , can be derived as                       I (i, j ), if sd ...
Scale-dependent Saliency• A pixel may become indistinguishable from its  neighbors to a human observer.• An image patch th...
Scale-dependent Saliency• First, the original image is scaled in homogeneous  into the final thumbnail size, i.e. the disp...
Scale-dependent Saliency• the minimum recognizable pattern sd    • In our monitor : 1680X1050@65Hz , 120 DPI•      sd is...
Scale-dependent Saliency2012/6/19                              15
Objectness Measure• Preserve object completeness as much as possible• Alexe et al. What is an object? (CVPR,2010).      – ...
Objectness Measure• First sample nw windows W  {wi }iw1 for an input image• Then calculate the objectness map O as the a...
Scale and Object Aware Saliency            S S            so     scale                           (i, j ) O(i, j )2012/6/1...
Cyclic Seam Carving• In many cases a seam has no choice but to cross objects due  to the original definition of seam.• Dis...
Cyclic Seam Carving• Cyclic Seams: warp the image2012/6/19                         20
Cyclic Seam Carving• Energy function              Escale    Esc  (1   )  S so   where ρ is the weight and set to 0....
Cyclic Seam Carving2012/6/19                         22
Image Warping Function• Many discrete retargeting methods generate  excellent results in general but they sometimes  creat...
Thin-plate-spline(TPS)• Landmark points          P   pi      2                                              , i  1,2, ...
TPS Warping Function• Landmark point pairs P and Q is derived from the CSC  retarget algorithm• A two-way solution• First,...
TPS Warping Function• Then, point set Q is re-sampled uniformly on the  target image, which is generated by CSC.• Finally,...
Quantitative Experiments• To find the thumbnail that matches the  description• From 10 x 10 image thumbnails• Different me...
Quantitative Experiments2012/6/19                              28
Quantitative Experiments2012/6/19                              29
Qualitative Experiments• In general our method performs the best regarding  the (foreground) object size in the thumbnails...
RetargetMe2012/6/19    31
Limitations• Saliency distribution is scattered• Object is to big2012/6/19                              32
Conclusion• A scale and object aware image retargeting  method for thumbnail browsing.• Several new techniques      – Scal...
Upcoming SlideShare
Loading in …5
×

Scale and object aware image retargeting for thumbnail browsing

517 views

Published on

《Scale and Object Aware Image Retargeting for Thumbnail》的PPT
perillaroc@BIT

Published in: Technology, Business
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
517
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
16
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Scale and object aware image retargeting for thumbnail browsing

  1. 1. ICCV 2011 Scale and Object Aware ImageRetargeting for Thumbnail Browsing Jin Sun, Haibin Ling Temple University, Philadelphia, PA, 19122 By perillaroc
  2. 2. Introduction• Image browsing tasks• Tiny thumbnails: a fixed small size• Scaling – bring difficulties in searching and recognition• Automatic image retargeting methods – target size is comparable to that of the original image• Several important issues for thumbnail browsing2012/6/19 2
  3. 3. Thumbnail Browsing• Thumbnail scales – Studies have shown that scales can have significant effects on human visual perception• Object completeness – Low-level gradient-based information – NOW! The object-level completeness• Structure smoothness – The contamination caused by pixel removal methods, e.g. seam carving2012/6/19 3
  4. 4. Scale and Object Aware Retargeting• Thumbnail scales – An image perceived by human vision system – A new scale dependent saliency• Object completeness – integrating the objectness measurement recently proposed by Alexe et al.• Structure smoothness – Use the thin-plate-spline(TPS) as warping function – Use Cyclic seam Carving to guide the warping.2012/6/19 4
  5. 5. Scale and Object Aware Retargeting• A continuous retargeting algorithm but uses discrete retargeting to guide the warping transformation.2012/6/19 5
  6. 6. Overview : Task Formulation• Retargeting problem: J  F ( I , S ( I ))• Seam carving(SC)  I (i, j ), if j  seam(i ) F  sc  I (i, j  1), if j  seam(i ) i, j• First get retargeted images to a target size that is comparable to the original image size.• Then shrink the retargeted images to get the final thumbnails.2012/6/19 6
  7. 7. Overview : Framework Overview• Scale and object aware image retargeting(SOAP): F so• Warping function: thin-plate-spline (TPS)• Landmark points : cyclic seam carving (CSC)• Scale dependent saliency S scale ( I )• Object awareness O( I )• Scale and object aware saliency S so ( I )  S scale ( I )  O( I )• SOAP J  F ( I , S ( I )) so so2012/6/19 7
  8. 8. Flow chart of the proposed method2012/6/19 8
  9. 9. Scale-dependent Saliency• Original image size• Display size so s D sd  , s p  d rp• Projection size DPI D• DPI(Dots Per Inch)2012/6/19 9
  10. 10. Scale-dependent Saliency• How well such a system preserves the image information?• We want to make the foreground object/ content/theme of the image as “clear” as possible• According to the study, not all patterns in an image are recognizable by human2012/6/19 10
  11. 11. Scale-dependent Saliency• the perceived image, denoted as I p , can be derived as  I (i, j ), if sd / D  [ ,  ] I p ( I (i, j ))    I N (i, j ), otherwise• I N (i, j ) is the mean value of N-connected neighbors of pixel I(i,j)• N is determined by display device specifications.• κ and ρ are the lower and upper bounds(in cycles per degree) respectively of human visual acuity.• κ and ρ define the limits at which the visual stimuli frequency becomes too low or too high to be recognized by human.• We use κ = 0.0175 and ρ = 0.83.2012/6/19 11
  12. 12. Scale-dependent Saliency• A pixel may become indistinguishable from its neighbors to a human observer.• An image patch that was salient in the original image may not appear salient to a human observer.• Scale dependent saliency2012/6/19 12
  13. 13. Scale-dependent Saliency• First, the original image is scaled in homogeneous into the final thumbnail size, i.e. the display size, 60x60 pixels in our experiment.• Then, the minimum recognizable pattern, denoted by  sd is determined by Eqn.4.• Finally, the scale dependent saliency S scale ( I ) is defined as S scale ( I )  S ( I p ) where S(.) is the standard saliency (Itti 1998)2012/6/19 13
  14. 14. Scale-dependent Saliency• the minimum recognizable pattern sd • In our monitor : 1680X1050@65Hz , 120 DPI•  sd is in average 0.009 inches.• The size is approximately the distance between two pixel lines on the screen, i.e. N = 4 in Eqn.5.• As a result, in the final thumbnail four adjacent neighbors of one image pixel patch with value differences in color space within certain threshold will be assigned their mean value, which means those pixels are unable to be distinguished by human.• Differences : 50 in our experiment2012/6/19 14
  15. 15. Scale-dependent Saliency2012/6/19 15
  16. 16. Objectness Measure• Preserve object completeness as much as possible• Alexe et al. What is an object? (CVPR,2010). – A novel objectness measure, which is trained to distinguish object windows from background ones. – For a rectangular window w = (i,j,w,h) with a top-left corner at (i,j), width w and height h, its objectness is defined as the probability p(w) that w contains an object.2012/6/19 16
  17. 17. Objectness Measure• First sample nw windows W  {wi }iw1 for an input image• Then calculate the objectness map O as the average objectness reponse at each pixel 1 O(i, j )  , j )w pobj (w)  wW (i Where   max O(i, j ) i, j• nw = 10000 in our experiments2012/6/19 17
  18. 18. Scale and Object Aware Saliency S S so scale (i, j ) O(i, j )2012/6/19 18
  19. 19. Cyclic Seam Carving• In many cases a seam has no choice but to cross objects due to the original definition of seam.• Discontinuous seams• Discontinuous seam-carving for video retargeting. CVPR, 2010• Problem: object-structure damage, e.g. pixel shift problem2012/6/19 19
  20. 20. Cyclic Seam Carving• Cyclic Seams: warp the image2012/6/19 20
  21. 21. Cyclic Seam Carving• Energy function Escale    Esc  (1   )  S so where ρ is the weight and set to 0.3 .• The improved energy is then combined with the CSC algorithm to provide landmark point pairs needed for estimating TPS warping.2012/6/19 21
  22. 22. Cyclic Seam Carving2012/6/19 22
  23. 23. Image Warping Function• Many discrete retargeting methods generate excellent results in general but they sometimes create serious artifacts when the target has a size much smaller than the input.• We combine a continuous warping model with a discrete retargeting guidance.• Thin-plate-spline(TPS) as our warping function2012/6/19 23
  24. 24. Thin-plate-spline(TPS)• Landmark points P   pi  2 , i  1,2, and , nl  Q  qi  2 , i  1,2, , nl  , where p is mapped to q.• the TPS transformation T is defined as the transformation from P to Q that minimizes the regularized bending energy E(f) defined as E ( f )   qi  f (qi )  2 i 2 f 2 2 f 2 2 f 2   ( 2 )  2( )  ( 2 ) dxdy x xy y• The TPS warping is defined as T  arg min f E ( f )2012/6/19 24
  25. 25. TPS Warping Function• Landmark point pairs P and Q is derived from the CSC retarget algorithm• A two-way solution• First, We sample randomly a landmark set b P (Fig. 6(a)) from original image and then trace their shifting in the CSC process.2012/6/19 25
  26. 26. TPS Warping Function• Then, point set Q is re-sampled uniformly on the target image, which is generated by CSC.• Finally, a sample set P is generated by mapping Q to the original image using warping estimated by Q and P.2012/6/19 26
  27. 27. Quantitative Experiments• To find the thumbnail that matches the description• From 10 x 10 image thumbnails• Different methods – Scaling(SL) – Seam carving (SC) – Improved seam carving (ISC) – The proposed SOAR algorithm (SOAR)2012/6/19 27
  28. 28. Quantitative Experiments2012/6/19 28
  29. 29. Quantitative Experiments2012/6/19 29
  30. 30. Qualitative Experiments• In general our method performs the best regarding the (foreground) object size in the thumbnails.2012/6/19 30
  31. 31. RetargetMe2012/6/19 31
  32. 32. Limitations• Saliency distribution is scattered• Object is to big2012/6/19 32
  33. 33. Conclusion• A scale and object aware image retargeting method for thumbnail browsing.• Several new techniques – Scale dependent saliency – Objectness – Cyclic seam carving – A TPS-based continuous warping model2012/6/19 33

×