• Like
  • Save
study Seam Carving For Content Aware Image Resizing
Upcoming SlideShare
Loading in...5

study Seam Carving For Content Aware Image Resizing







Total Views
Views on SlideShare
Embed Views



1 Embed 3

http://www.slideshare.net 3



Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
Post Comment
Edit your comment
  • Shai Aidan 以色列人MERL(Mitsubishi Electric Research Labs)沒想到以色列人投效到日本研究中心

study Seam Carving For Content Aware Image Resizing study Seam Carving For Content Aware Image Resizing Presentation Transcript

  • Seam Carving for Content-Aware Image Resizing
    Shai Aidan (Mitsubishi Electric Research Labs)
    Ariel Shamir (The Interdisciplinary Center & MERL)
  • Resize
    Seam carving & insertion
  • Abstract
    Seams are optimal 8-connected paths of pixels cross the image
    Carving out or inserting seams to achieve content-aware resizing
  • Outline
    Seam-carving operator
    Discrete image resizing
    Multi-size images
    Conclusions and future work
  • Introduction
  • Motivation
    HTML can support dynamic changes of page layout and text. Why can not an image deform to fit different layout automatically ?
    How about aspect ratio of an image , such as fitting photo into PDA or phone cells ?
    Solution ?
    Resize – content independent
    Crop – remove pixels from the image periphery only
  • Basic Idea of Seam-Carving
    Use energy function to define the importanceof pixels
    Define seam-carving image operator
    Image reduction
    Carving out seams - the connected low energy pixels crossing the image
    Preserving the image structure
    Image enlarging
    Insert seams on low energy area
    The order of seam insertion ensures a balance between the original image content and the artificially inserted pixels
  • Application
    Discrete image resizing
    Aspect Ration Change, Image Retarget, Image Enlarging, Content Amplification, Seam Carving in gradient domain, Object Removal
    Multi-size images
    An image can continuously change their size in a content-aware manner
    Storing the order of seam removal and insertion
  • background
  • Image Retarget
    Seek to change the size of the image while maintaining the important features
    Face detector
    An automatic thumbnail creation [Suh03]
    Fisheye-View warp [Liu and Gleicher 05, 06]
    Visual saliency []
    [Suh 03]
    [Selur 04, decompose image to foreground obj and background
  • Saliency map
    [Itti IEEE99]
    Simulate neuroscience of human visual system
    Pyramid tech. to compute 3 feature maps, color, intensity and orientation
    [Suh 03], an automatic thumbnail creation, based on either a saliency map or the output of a face detector
    [Chen 03], adapting most important region of images to mobile devices.
  • [Liu 03], suggesting to trade time for space. Given a collection of regions of interest, they construct an optimal path through these regions and display them serially.
  • [Santella et al. 06] use eye tracking, in addition to composition rules to crop images intelligently.
  • ROI (Region-Of-Interest)
    Such a method was proposed by [Liu and Gleicher 05, 06] for image and video retargeting. For image retargeting they find ROI and construct a novel Fisheye-View warp that essentially applies a piecewise linear scaling function in each dimension to the image. This way the ROI is maintained while the rest of the image is warped. The retargeting can be done in interactive rates, once the ROI is found, so the user can control the desired size of the image by moving a slider. In their video retargeting work they use a combination of image and saliency maps to find the ROI. Then they use a combination of cropping, virtual pan and shot cuts to retarget the video frames.
  • Feature-aware warping
    The first solution to the general problem of warping an image into an arbitrary shape while preserving user-specified features was recently proposed by [Gal et al. 06].
    The feature-aware warping is achieved by a particular formulation of the Laplacian editing technique, suited to accommodate similarity constraints on parts of the domain.
    Since local constraints are propagated by the global optimization process, not all the constraints can always be satisfied at once
  • Seam
    Perfect seams to combine parts of a set of photo into a single composite picture [Agarwala et al. 04]
    Drag-and-Drop Pasting that extends the Poisson Image Editing to computer an optimal boundary (seam) between the source picture and target images [Jia et al. 06]
    AutoCollage, a program that automatically creates a collage image from a collection of images. [Rother et al. 06]
    Simultaneously solve matting and compositing. They allow the user to scale the size of the foreground object and paste it back on the original background. [Wang , Cohen 06]
    evaluated several cost functions for seamless image stitching and concluded that minimizing an L1error norm between the gradients of the stitched image and the gradients of the input images performed well in general [Zomet et al. 05]
  • Sear Optimal Seam
    Dijkstra’s shortest path algorithm [98]
    Dynamic programming [Efros 01]
    Graph cuts [Kwatra 03]
  • Seam-carving operator
  • Strategies of Image Reduction
    e1 energy
    global remove the lowest energy pixels
    remove the least energy in each row
  • Strategies of Image Reduction
    e1 energy
    removing columns with minimal energy
    find a sub-win with the highest energy
  • Vertical Seam
  • Horizontal Seam
  • Optimal Seam Search
  • Optimal Seam Search
    Dynamic Programming
  • e1 energy
  • Image Energy Preservation
    The average energy of all pixels during resizing
  • Energy Functions
    L1 and L2-norm of the gradient, saliency measure [Itti 99]
  • Histogram of Gradient (HoG)
    • Histogram of Gradient (HoG) [Dalal and Triggs 95]
    Dividing the image window into cells
    For each cell accumulating a local 1-D histogram of gradient directions
    Normalize cells by the measure of local histogram energy over larger blocks
    The average gradient image
    Weighted R-HOG descriptor
    R-HOG descriptor
  • Energy Functions
    Histogram of Gradient (HoG) [Dalal and Triggs 95]
    max(HoG(I(x,y)) makes sure the seams run parallel to the edge of objects and not cross them
  • Energy Functions
    Compute the entropy over a 9 x 9 window and add it to e1
    eEntropy(x,y) =
    + e1 (x,y)
  • Energy Functions
    Segmentation and L1
    Image segmentation [Christoudias 02]
    Apply e1 on the results
  • No single e function performs well across all images
    Similar range for resizing
    e1 or eHoG works well
  • Discrete image resizing
    Aspect Ratio Change, Retargeting with Optimal Seams-Order, Image Enlarging, Content Amplification,
  • Aspect Ratio Change
    Carving-out /insert seams
    1D aspect ratio changing
  • 2D aspect ratio changing
    Optimal Seams-Order Search
  • Retargeting with Optimal Seams-Order
    Transport map
  • Image Enlarging
    Find first k seams for removal
    Duplicate them in order to arrive at I(-k)
    I(t): smaller image after t seam-carving
    I(-k): enlarged image after k seam insertion
    enlarged image
    insert seams in order of removal
  • Image Enlarging (>50%)
    Break into several steps
    Each step does not enlarge the size of image more than a fraction
  • Content Amplification
  • Seam Carving in the Gradient Domain
    Seam + Poisson Reconstruction [Perez 03]
    Compute e function
    Work on the gradient domain
    Remove seams from the x and y derivatives of the original image
    Use Poisson Reconstruction
    retarget in
    Gradient Domain
  • Object Removal
    Mark the removing target
    Remove seams until all the marked pixels are gone
    * Employ seam insertion to maintain the original size
  • Object Removal
  • Multi-size images
    Store the pre-computed representation that encodes, for each pixel in V/H map
    The index of the seam that removed it
    The negative index of the seam that inserted it
    Blue (first seam)  Red (last seam)
    V(i,j)=t : pixel (i,j) removed by t-th vertical seam
    H(i,j)=t : pixel (i,j) removed by t-th horizontal seam
  • Limitations
    Seam-Carving does not work well on all images
    Ex: face
    Constraint the face
    Face the flower
    Bottom up feature detection
  • Limitations
    The amount of content
    Too density, no “less” important area
    The layout of the image content
  • Conclusions
    Present a content-aware resizing using the seam-carving image operator
    Seams are the optimal paths on a single image
    Carve-out seams
    Insert seams
    Application of seam-carving operator
    Aspect ratio change, image retargeting, content amplification, object removal
    Multi-size images that support continuous resizingin real-time
  • Future Work
    Video resizing
    Combination of scaling and seam-carving
    Define more robust multi-size image
    Better solution to combine horizontal and vertical seams in multi-size image
  • END