study Image Vectorization using Optimized Gradeint Meshes

Image Vectorization using Gradient Meshes
Jian Sun, Lin Liang, Fang Wen, Heung-Yeung Shum
Microsoft Research Asia
ACM SIGGRAPH 2007

Cutout tool
Initial mesh
Input image
Optimized gradient mesh
Reconstruction

Outline
Introduction
Background
Gradient Mesh
Optimized Gradient Mesh
Result
Conclusions

Introduction

Image Vectorization
Goal : convert a raster image into a vector graphics
Compact
Scalable
Easy to animate
Requirements
Vector-based contents (eg. Flash or SVG) on the Internet
Vector-based GUIs used in Windows Vista

Gradient mesh
Gradient mesh, adrawing tool of commercial vector graphics editors
Tracing photograph
Start adding mesh points
Selecting mid value skin tone
Sampling colors from face mesh to hide seam
Sampling colors from photo
Sampling colors from within the mesh
Finished eye/eye socket
http://www.creativebush.com/tutorials/mesh_tutorial.php

Image represented by gradient mesh
gradient mesh
http://www.creativebush.com/tutorials/mesh_tutorial.php

Image vectorization tools
Adobe Illustrator, “Live Trace”
Corel CoreDraw, “CorelTrace”
AutoTrace, “AutoTrace”
Input image
Adobe, Live Trace

Blend surface colors according to the control points color as constructing surface by the control points
Optimize the gradient mesh as an energy minimization problem
Advantages
Efficiency of use
Easy to edit – modify, animation
Scalability
Compact representation
JPEG, 37.5 KB
Optimized, 7.7KB

Background

Object-based vectorization
Object-based vectorization [Price and Barrett 06]
Hierarchically segmentation of object and sub-objects by a recursive graph cut algorithm
Subdivide meshes until the reconstruct error is below a threshold
Input image
Subdivision mesh

RaveGrid
[Swaminarayan and Prasad 06]
Constrained Delaunay triangulation of the edge contour set

cont.

Ardeco
Automatic Region Detection and Conversion algorithm [Lecot and Levy 06]
Cubic splines
Each region filled with a constant color, or a linear or circular gradient

cont.

Gradient Mesh

Overview
Cutout tool
Input raster image
Initial mesh
(Coons mesh)
E(M) Constrains:
Smoothness
User guidedvector
Boundary
min argEnergy(Mesh)
non-linear least squares (NULL) problem
Levenberg-Marquardt (LM) algorithm
(Ferguson patch)
Reconstruction image

Color fitting: monotonic cubic spline interpolation
Coherent matting method: sample object boundary color from the estimated foreground colors

Mesh initialization (manually)
Decompose the input image into sub-objects using Cutout tool [Li et al. 04]
Divide each sub-objects with 4 segments manually as Coons patches
m(u,v): the position vector of a point (u,v)
Q : control points el al.
F : basis functions

Ferguson patch
TA
TB
[Ferguson 1964]
P(0)=A
Basic Curve Segmentation
P(1)=B

Ferguson patch
Basic Surface Segmentation
0

Gradient mesh
A gradient mesh consists of topologically planar rectangular Ferguson patches with mesh-lines
For each point q

Position: {xq,yq}

Derivatives: {mqu,mqv, αqumqu, αqvmqv}
RGB color: cq = {cq(r), cq(g), cq(b)}

Ferguson patches are lack of Cv and Cu !
Color interpolation

[Wolberg and Alfy 99]
Determine the smoothest possible curve that passes through its control points and satisfy monotonic constraint
The seven data points are monotonically increasing in f(xi) for 0 ≦i ≦ 6, the cubic spline is not monotonic
Monotonic cubic spline

Rendering Ferguson patches
Sample color of control points
Estimate Cu, Cv by Monotoic Cubic Spline algorithm
Render Ferguson patches

Scalability
A gradient mesh

original resolution (x 1)

Scaling result (x8)

shape edges are well preserved

Bi-cubic raster scaling (x8)

Blocky artifacts appear

Minimize E(M)
min arg.

Solve NULL problem using LM algorithm
Minimizing E(M) is a non-linear least squares (NULL) problem
Energy function
z: vector form of unknowns in M
Levenberg-Marquardt (LM) algorithm is the most successful solver for NULL
[Levenberg 44], [Nocedal and Wright 99]

cont.

Gaussian pyramid from the input image and apply coarse-to-fine optimization for LM

Optimization
Gradient mesh of Adobe Illustrator

0.7/pixel reconstruction error
40 iterations

Smooth constraint
Opt. gradient mesh without smooth(err. 1.8/pixel)
Opt. gradient mesh with smooth (err. 0.9 pixel)
Input image
Smooth neighboring patches also,
mp(- △s, t) = mp-1(1- △s, t)

Vector line guided optimized gradient mesh
User guided vector, V
Initial mesh
Opt. gradient mesh with user guided vector
(err. 0.5/pixel)
Directly optimized gradient mesh
(err. 2.5/pixel)

Vector line guided optimized gradient mesh
w = 1/5 L
Initial mesh
Opt. gradient mesh with V

Boundary constraint
The boundary of a gradient – one or more cubic Bezier spline
The control points on the boundary only move along the spline
Ex: control point q on the spline S in u direction

Results

Red pepper
Optimized
the highlight and shadow regions are reconstructed
Initial
gradient meshes
Gradient meshes by an artist (354 patches)

Sculpture
Input image
Reconstruction

Face
Input image
Reconstruction

Conclusions
Input image
Introduce the gradient mesh as an image representation tool first
Present optimized gradient mesh
Limitations
A fine image details and highly textured image
Boundaries or topologies are too complicated
Reconstructed image
Optimized gradient meshes

END