study Seam Carving For Content Aware Image Resizing
1. Seam Carving for Content-Aware Image Resizing Shai Aidan (Mitsubishi Electric Research Labs) Ariel Shamir (The Interdisciplinary Center & MERL) ACM SIGGRAPH 2007
6. Motivation HTML can support dynamic changes of page layout and text. Why can not an image deform to fit different layout automatically ? iGoogle How about aspect ratio of an image , such as fitting photo into PDA or phone cells ? Solution ? Resize – content independent Crop – remove pixels from the image periphery only
7. Basic Idea of Seam-Carving Use energy function to define the importanceof pixels Define seam-carving image operator Image reduction Carving out seams - the connected low energy pixels crossing the image Preserving the image structure Image enlarging Insert seams on low energy area The order of seam insertion ensures a balance between the original image content and the artificially inserted pixels
8. Application Discrete image resizing Aspect Ration Change, Image Retarget, Image Enlarging, Content Amplification, Seam Carving in gradient domain, Object Removal Multi-size images An image can continuously change their size in a content-aware manner Storing the order of seam removal and insertion
10. Image Retarget Seek to change the size of the image while maintaining the important features Face detector An automatic thumbnail creation [Suh03] ROI Fisheye-View warp [Liu and Gleicher 05, 06] Visual saliency [] [Suh 03] [Selur 04, decompose image to foreground obj and background origin
11. Saliency map [Itti IEEE99] Simulate neuroscience of human visual system Pyramid tech. to compute 3 feature maps, color, intensity and orientation [Suh 03], an automatic thumbnail creation, based on either a saliency map or the output of a face detector [Chen 03], adapting most important region of images to mobile devices.
12. [Liu 03], suggesting to trade time for space. Given a collection of regions of interest, they construct an optimal path through these regions and display them serially.
13. [Santella et al. 06] use eye tracking, in addition to composition rules to crop images intelligently.
14. ROI (Region-Of-Interest) Such a method was proposed by [Liu and Gleicher 05, 06] for image and video retargeting. For image retargeting they find ROI and construct a novel Fisheye-View warp that essentially applies a piecewise linear scaling function in each dimension to the image. This way the ROI is maintained while the rest of the image is warped. The retargeting can be done in interactive rates, once the ROI is found, so the user can control the desired size of the image by moving a slider. In their video retargeting work they use a combination of image and saliency maps to find the ROI. Then they use a combination of cropping, virtual pan and shot cuts to retarget the video frames.
15. Feature-aware warping The first solution to the general problem of warping an image into an arbitrary shape while preserving user-specified features was recently proposed by [Gal et al. 06]. The feature-aware warping is achieved by a particular formulation of the Laplacian editing technique, suited to accommodate similarity constraints on parts of the domain. Since local constraints are propagated by the global optimization process, not all the constraints can always be satisfied at once
16. Seam Perfect seams to combine parts of a set of photo into a single composite picture [Agarwala et al. 04] Drag-and-Drop Pasting that extends the Poisson Image Editing to computer an optimal boundary (seam) between the source picture and target images [Jia et al. 06] AutoCollage, a program that automatically creates a collage image from a collection of images. [Rother et al. 06] Simultaneously solve matting and compositing. They allow the user to scale the size of the foreground object and paste it back on the original background. [Wang , Cohen 06] evaluated several cost functions for seamless image stitching and concluded that minimizing an L1error norm between the gradients of the stitched image and the gradients of the input images performed well in general [Zomet et al. 05]
29. Energy Functions Histogram of Gradient (HoG) [Dalal and Triggs 95] max(HoG(I(x,y)) makes sure the seams run parallel to the edge of objects and not cross them
30. Energy Functions Entropy Compute the entropy over a 9 x 9 window and add it to e1 eEntropy(x,y) = + e1 (x,y)
37. Image Enlarging Find first k seams for removal Duplicate them in order to arrive at I(-k) origin I(t): smaller image after t seam-carving I(-k): enlarged image after k seam insertion t I(-1) I(-k) I(t) I(-k) enlarged image insert seams in order of removal
38. Image Enlarging (>50%) origin Break into several steps Each step does not enlarge the size of image more than a fraction
40. Seam Carving in the Gradient Domain Seam + Poisson Reconstruction [Perez 03] Compute e function Work on the gradient domain Remove seams from the x and y derivatives of the original image Use Poisson Reconstruction original retarget retarget in Gradient Domain
41. Object Removal Mark the removing target Remove seams until all the marked pixels are gone * Employ seam insertion to maintain the original size
43. Multi-size images Store the pre-computed representation that encodes, for each pixel in V/H map The index of the seam that removed it The negative index of the seam that inserted it Blue (first seam) Red (last seam) origin V(i,j)=t : pixel (i,j) removed by t-th vertical seam H(i,j)=t : pixel (i,j) removed by t-th horizontal seam
44. Limitations Seam-Carving does not work well on all images Ex: face Origin Crop Scale Constraint the face Face the flower Bottom up feature detection
45. Limitations The amount of content Too density, no “less” important area The layout of the image content origin origin
46. Conclusions Present a content-aware resizing using the seam-carving image operator Seams are the optimal paths on a single image Carve-out seams Insert seams Application of seam-carving operator Aspect ratio change, image retargeting, content amplification, object removal Multi-size images that support continuous resizingin real-time
47. Future Work Video resizing Combination of scaling and seam-carving Define more robust multi-size image Better solution to combine horizontal and vertical seams in multi-size image