This explains the main points taken from Cootes et all papers for ASM and AAM models.

1. We value our relationship
We value our relationships.
9 December 2012
© IRDC India 2012 www.irdcindia.comWe value our relationship
Chandrashekhar Padole
Title for PresentationActive Shape/Appearance Model ( ASM & AAM)
by
IRDC India

2. IRDC India
www.irdcindia.com
Presenter: IRDC India
info@irdcindia.com
9 December 2012 2
Objectives
• To understand Active Shape Model
• To investigate the computations involved in ASM
• MATLAB Modules
• To understand Active Appearance Model
• To investigate the computations involved in AAM
• Application in Pose Estimation and Pose Compensation

3. IRDC India
www.irdcindia.com
Presenter: IRDC India
info@irdcindia.com
9 December 2012 3
Roadmap
•Paper I
•Paper II
•Paper III
• Paper IV
• Paper V

4. IRDC India
www.irdcindia.com
Presenter: IRDC India
info@irdcindia.com
9 December 2012 4
Paper I
Title: Training Models of shape from set of examples
Authors:T.E. Cootes, C.J.Taylor et.al (BMVC 1992)
•Shape of resistors
–different shapes in
some extent
•Labelling training set

5. IRDC India
www.irdcindia.com
Presenter: IRDC India
info@irdcindia.com
9 December 2012 5
Contd..
•Constraints over labelling point across the set of training samples
• Point 0 & 31 always represent end of wires
• Point 3 , 4 ,5 represent one end of the body of the resistor and so on
• Manual process is ok for simple shapes but for some biological complex
shapes automated tools
•Alligning the training set
• By scaling, rotating and translating
• To make them as close as possible – minimise a weighted sum of suqares of
distances between equivalent points of diffrent shapes
• This is form of Genralized Procrustes Analysis

6. IRDC India
www.irdcindia.com
Presenter: IRDC India
info@irdcindia.com
9 December 2012 6
Contd..
•Genralized Procrustes Analysis ( Ref- Generalized Procrustes
analysis by Gower , 1975)
• Let xi be a vector describing the n points of the ith shape in the set
– xi=(xi0,yi0, xi1,yi1 ,...... xik,yik ........ xi(n-1),yi(n-1)
• Let Mj[xj] be a raotaion by ϴj and a scaling by sj
• Given two similar shapes ,xi and xj, we can choose ϴj , sj and translation
(tx,ty)j mapping xi onto Mj[xj] so as to minimize weighted sum
where
and W is diagonal matrix of weights for each point and used
for giving importance more or less to corresponding lable
points and it can be choosen practially as described ahead
])[.(.])[( jji
T
jjij xMxWxMxE −−=








+−
+−
=







jyjkjjjkjj
jxjkjjjkjj
jk
jk
j
tysxs
tysxs
y
x
M
)cos()sin(
)sin()cos(
θθ
θθ

7. IRDC India
www.irdcindia.com
Presenter: IRDC India
info@irdcindia.com
9 December 2012 7
Contd..
•Choosing Wk
• let Rkl be the distance between points k and l in a shape
• VRkl
be the varaince in this distances over the set of shapes, then Wk for the kth
point ,
• Thus ,a point tends to be remain fixed with respect to the otehr , if sum of variances
will be small , a large weight will be given and matching such points indiffrent
shapes will be a priority
11
0
−−
=






= ∑
n
l
Rk kl
VW

8. IRDC India
www.irdcindia.com
Presenter: IRDC India
info@irdcindia.com
• Alignment algorithm
1. Rotate ,scale and trasnlate each of the shapes in the set to align the first
shape
2. Repeat
1. Calcualte the mean of the transformed shapes
2. Either
1. Adjust the eman to a default scale ,orientation and origin
2. Rotate scale translate the mean to align the first shape
3. Rotate ,scale and translate each of the shapes again to amtch the adjusted mean
3. Untill Convegence
•Inside the iteration loop , it is requried to renormalise the
mean,without whcih this algorithm is ill-conditioned
Practical Implementation:
9 December 2012 8
Contd..

9. IRDC India
www.irdcindia.com
Presenter: IRDC India
info@irdcindia.com
9 December 2012 9
Contd..
•Statistics of set of aligned shapes
• Mean shape
• Apply PCA to deviations from
mean to find the modes of variations
∑=
=
sN
i
i
s
x
N
x
1
1

10. IRDC India
www.irdcindia.com
Presenter: IRDC India
info@irdcindia.com
9 December 2012 10
Paper II
Title: Use of Active Shape Models for locating structures in Medical
Images
Authors:T.E. Cootes, C.J.Taylor (Image & Vision Computing 1994)
•Modelling object shape-point distribution model ( PDM)
• We have set of images containing examples of variable structure
• E.g. Left Ventricle –shape of this can vary bioth with time( as heart beats)
and across individuals
• These shape variations is to be modeled ( ASM)
• Choose points around the left ventricle boundary and also around the nearby
edge of the right ventricle and top of the left atrium

11. IRDC India
www.irdcindia.com
Presenter: IRDC India
info@irdcindia.com
9 December 2012 11
Contd..
•Addition of points other than left ventricle gives more specific
model whcih peroform better iamge search for left ventricle
•Each smaple from set of examples , represented by 96 points
•Used 66 images ( examples/samples)
•11 key positions were marked on each boundary
•96 points points gnerated from the key points along the boundaries
between key positionss
•In order to be able to compare equivalent points from diffrent
shapes, they were alligned by scaling, rotating and translating the
training shapes so that they correspond as closely as possible

12. IRDC India
www.irdcindia.com
Presenter: IRDC India
info@irdcindia.com
9 December 2012 12
Contd..
•Capturing the statistics of a set of aligned shapes
• N aligned shapes, then ith shape is given by
– xi=(xi0,yi0, xi1,yi1 ,...... xik,yik ........ xi(n-1),yi(n-1)
• Mean shape and deviation
• 2n x 2n covaraince matrix S ,
• Eigen Analysis of S will give modes of variations by Pk(k=1....2n)
Where is kth eigen value of S and ,
• t modes of variations – select t columns of Pk corresponding values
of
∑=
=
N
i
ix
N
x
1
1
xxdx ii −=
∑=
=
N
i
T
iidxdx
N
S
1
1
kkp PS k
λ=
kλ 1+≥ kk λλ 1=k
T
k PP
kλ

13. IRDC India
www.irdcindia.com
Presenter: IRDC India
info@irdcindia.com
9 December 2012 13
Contd..
•Shape reconstruction:Any shape in training set can be apporximated
using mean shape and weighted sum of deveiations from first t
modes
where P=(P1,P2,...Pt) is the matrix of first t eigen vectors
and b = (b1,b2,...bt)T is the vector of weights
•Since, eigen vectors are orthogoanl , so
•The above equation allow us to generate new examples of the shapes
by varying the parameters (b)within suitable limits, so the new
shapes will be similar to those in the training set.
•The limits for each each element of b , bk, are derived by examining
the distribution of the parameters values required to generate the
training set.
Pbxx +=
1=PPT
)( xxPb T
−=

14. IRDC India
www.irdcindia.com
Presenter: IRDC India
info@irdcindia.com
9 December 2012 14
Contd..
•If gaussian distribution are assumed , one can choose sets of
parameters {b1,b2,...bt }such that the Malhalanobis distance (Dm)
from teh mean is less than sutiable value Dmax
•Effects of variations of model parameters,{b}
2
max
1
2
2
D
b
D
t
k k
k
m ≤





= ∑= λ

15. IRDC India
www.irdcindia.com
Presenter: IRDC India
info@irdcindia.com
9 December 2012 15
Contd..
•Modelling Grey Level Appearance
• Grey level patterns about the key point of model in images of diffrent
examples will be often similar
• For every model point i in each image j, we can extract a profile ,gij, of
length np pixels, centered at the point
• Author choose to smaple the derivative of the grey levels along the profile in
the image and normalize it.
• Profile runs from Pistart
to Piend
and is of length np pixels,
• kth element of the derivative profile is
where yik is the kth point along the ith profile and is given by
and is the grey level in image j at that point
)()( )1()1( −+ −= kijkijijk yIyIg
)(
1
1
startendstart ii
p
iik pp
n
k
py −
−
−
+=
)( ikj yI

16. IRDC India
www.irdcindia.com
Presenter: IRDC India
info@irdcindia.com
9 December 2012 16
Contd..
•We then normalise this profile vector,
•For each point ,i, we can calcualte a mean normalised derivative
profile
•Sgi is the npxnp covraince matrix of
•Calcualting the eigen vector and values , we get model parameter
associated with grey level or appearance.
∑=
=′
pn
k
ijk
ij
ij
g
g
g
1
∑=
′=
sN
j
ij
s
i g
N
g
1
1
ig

17. IRDC India
www.irdcindia.com
Presenter: IRDC India
info@irdcindia.com
9 December 2012 17
Contd..
Practical Implementation:

18. IRDC India
www.irdcindia.com
Presenter: IRDC India
info@irdcindia.com
9 December 2012 18
Paper III
Title: Modelling Object Appearance using the Grey-level Surface
Authors: T.E. Cootes, C.J.Taylor (BMVC 1994)
•Based on Landmark ( LM) points and triangulation
• Labeled point-particular part of stucture, a corner, a point of high intesity etc

19. IRDC India
www.irdcindia.com
Presenter: IRDC India
info@irdcindia.com
9 December 2012 19
Contd..
• First traingulate origianl LM points . Then Add traingualtion by
adding additioanl points at the mid-point of each connecting arc
•Instead of applying traingualtion to each example(sample), it is
applied to the mean configuration of landmarks. This mean is
generated by alligning the set of examples so that they overlap as
much as possible and then calculating the mean of each co-ordinate
for the LM point

20. IRDC India
www.irdcindia.com
Presenter: IRDC India
info@irdcindia.com
9 December 2012 20
Contd..
•To add the appearance infaotaion to the shape each point is
decribed as triplet. ( xi,yi,Ii) to have
x=(x0,y0,λI0 , x1,y1, λ I1 ...... Xn-1,yn-1, λ In-1 )
where λ proportionality constant to allow for x & y being
measured in diffrent units to the grey-level intensity.
•Apply PCA to the set of example vectors ,
the mean set of points
P 3n x t matrix, the columns of which are the t orthonramal
unit eigen vectors of the covaraince matrix corresponding to the
largest eighen value ( each column describes a mode of shape
varaition in the data, the first being the msot significant
b set of t model parameters
PbxX += ˆ
xˆ

21. IRDC India
www.irdcindia.com
Presenter: IRDC India
info@irdcindia.com
9 December 2012 21
Contd..
•Varying the model parameters , b=(b1,b2,...bt) with certain limits
whcih can be learnt from the stastics of the training set, we can
generate new examples of the 3D shape,simialar the those in training
set.
•Varying each parameter causes changes to both the position of each
landmark and the intesity values at that landmark.
•Example of models:
• Eye model
• Banana model

22. IRDC India
www.irdcindia.com
Presenter: IRDC India
info@irdcindia.com
9 December 2012 22
Contd..
•Eye Model-
• 10 example (left eye from 10 persons)
• 9 base landmark points
• Traingulation algorithm with two iterations
of interpolations to get 345 points
• the relative importance of intensity
to x,y values , λ=0.25,

23. IRDC India
www.irdcindia.com
Presenter: IRDC India
info@irdcindia.com
9 December 2012 23
Contd..
•Banana model-
• Bananas were
illuminated from
diffrent directions
• 33 points 369 points
• λ=0.25
•Application: Image
Search

24. IRDC India
www.irdcindia.com
Presenter: IRDC India
info@irdcindia.com
9 December 2012 24
Paper IV
Title: Active Shape Models : Evaluation of a Multi-resolution
Method for Improving Image Search
Authors: T.E.
Cootes, A. Lanitis, C.J.Taylor (BMVA 1994)
•Image Search using an Active Shape Model
Where
M(s,ϴ)[.] performs a rotation by ϴ and a scaling by s
xc,yc is the position of the centre of the model in the
image frame
• Image search problem searching for s, ϴ, (xc,yc)
T
ccccc
c
yxyxX
XxsMX
),.......,(
])[,(
=
+= θ

25. IRDC India
www.irdcindia.com
Presenter: IRDC India
info@irdcindia.com
9 December 2012 25
Contd..
•An iterative approach to improving the fit of the instance, X, to an
image proceeds as follows :
1. Examine a region of the image around each point to calculate the
displacement of the point required to move it to a better location.
2. From these displacements calculate adjustments to the pose and
the shape parameters.
3. Update the model parameters; by enforcing limits on the shape
parameters, global shape constraints can be applied ensuring the
shape of the model instance remains similar to those of the
training set.

26. IRDC India
www.irdcindia.com
Presenter: IRDC India
info@irdcindia.com
9 December 2012 26
Contd..
•To find a better location for each model point we sample a profile
perpendicular to the boundary at the point and run the grey-level
model along it to find best match
•Use least square appraoch to find the best change in pos
(dxc,dyc,ds,dϴ)

27. IRDC India
www.irdcindia.com
Presenter: IRDC India
info@irdcindia.com
9 December 2012 27
Contd..
• Problem- choosing the length of profile along which to search with
grey-level model
• If the search profile is too short , the model landmark points must be close
to their targets in teh iamge before they can ‘latch on ‘ and pull the shape
model into place.
• If they are too long the search becomes computationally expensive and the
grey-level models are more likely to latch on to distracting stractures in the
image away from the target object,preventing ASM from converging to the
correct shape.
•Solution: far from target,make large jumps and as model approaches
target structure, search should be restricted to immediate locality
• Multi resolution approach- Model to be applied first at coarse mean
to low resolution image , then refined oh higher resolution images.
• Low resolution images( Level 1 ,2...) can be obtained from Level 0,
i.e. original image by smoothing the image and subsampling every
other pixel.

28. IRDC India
www.irdcindia.com
Presenter: IRDC India
info@irdcindia.com
9 December 2012 28
Contd..
•As smoothing may modify the image structure, grey level model is
not restrinceted to be obtained only from Level 0 image but from
other levels of images.
•Thus, each landmark point will have set of grey level models.
•Start first from highest level image ( level N)and run number of
iterations of the ASM using the models trained at that level. Then
move to next level towards level 0.
•Note: higher level coarse and level 0 finest
•Examples of Multi resolution search
• Face Model
• Vertebra Model

29. IRDC India
www.irdcindia.com
Presenter: IRDC India
info@irdcindia.com
9 December 2012 29
Contd..
•Face Model- 169 points, 11 images of single person’s face,
greylevel models 7 pixel long
•Each iteration at
any level takes less
than 150ms on
Sun Sparc10
workstation
•Search completed
in 7 sec

30. IRDC India
www.irdcindia.com
Presenter: IRDC India
info@irdcindia.com
9 December 2012 30
Paper IV
Title: A Unified Approach to Coding and Interpreting Face Images
Authors: T.E. Cootes, G.J. Edwards, C.J.Taylor (ICCV 1995)
•Shape model-152 points on each of 160 training examples. 16
parameters

31. IRDC India
www.irdcindia.com
Presenter: IRDC India
info@irdcindia.com
9 December 2012 31
Contd..
•Shape-free Appearance Model
• Apply warping algortihm to deform all training images to the mean shape,
in such a way that changes in grey-level intensities are kept to a minimum
• Training images were deformed to the mean shape and grey level intensities
within the face area were
extracted.
• Each training example was
represented a vector
containing the nomalised
grey-level at each pixel in
the patch
• A flexible grey-level model
was generated for dataase;
only 79 parameters were
needed to explain 95% of
the variation

32. IRDC India
www.irdcindia.com
Presenter: IRDC India
info@irdcindia.com
9 December 2012 32
Contd..
•Local Appearance Model- 4 model parameters to explain 95% of
the variation

33. IRDC India
www.irdcindia.com
Presenter: IRDC India
info@irdcindia.com
9 December 2012 33
Contd..

34. IRDC India
www.irdcindia.com
Presenter: IRDC India
info@irdcindia.com
9 December 2012 34
Contd..

35. IRDC India
www.irdcindia.com
Presenter: IRDC India
info@irdcindia.com
9 December 2012 35
Contd..

36. IRDC India
www.irdcindia.com
Presenter: IRDC India
info@irdcindia.com
9 December 2012 36
Contd..

37. IRDC India
www.irdcindia.com
Presenter: IRDC India
info@irdcindia.com
9 December 2012 37
Paper V
Title: Statistical Models of Face Images –Improving Specificity
Authors: T.E. Cootes, C.J.Taylor et.al (IVC 1998)
•Previous work, Shape model and grey level model
•When new image is
presented to this
system, facial features
are located autoamtically
using active shape model
( ASM).
•The resulting
auomatically located
model points are
transformed into shape model parameters. The face is deforemd to the mean face
shape and grrey level appearance is transformed into the parameters of the shape-
free grey level model.

38. IRDC India
www.irdcindia.com
Presenter: IRDC India
info@irdcindia.com
9 December 2012 38
Contd..
•However, shape and grey-level varaintions may be correlated
• E.g. The shape mode of variation responsible for opening and clising the
mouth is correlated with the grey level mode responsible for appearance of
teeth
•Also , certain combinations of shape and grey-level modes may
correspond to illigal facial reconstruction
•Solution: Combine ( Shape+Grey level) = Appearance Model
• Train indvidually shape and shape free grey-level models
• Represent each training example by a vector containing both sape and
grey-level parameters.
• PCA is applied to these new training vectors in order to extract the
combined shape and grey level modes of variation.
• Before applying the final PCA , we scale the shape parameters so that their
varaince wihtin the training set is equal to the variance of the grey level
parameters.

39. IRDC India
www.irdcindia.com
Presenter: IRDC India
info@irdcindia.com
9 December 2012 39
Contd..
•Figure 3 below shows the first few modes of the combined
shape/grey-level model trained using image from Home Office DB.
•Figure 4 shows the parametric reconstructions of original images
using a compbined shape and grey-level model ( included hair)

40. IRDC India
www.irdcindia.com
Presenter: IRDC India
info@irdcindia.com
9 December 2012 40
Contd..
•Isolating sources of Variation
• To be done later
•Using Non-linear Shape Models
• To be done later

41. IRDC India
www.irdcindia.com
Presenter: IRDC India
info@irdcindia.com
9 December 2012 41
Paper VI
Title: Active Appearance Models
Authors: T.E. Cootes, G.J. Edwards, C.J.Taylor (PAMI June 2001)
•Statistical appearacne models are generated by combining a model
shape varaition with model of teture variation
• By “texture” means the pattern of inteisities or color across an image patch
• To build a face model , we require face iamges marked with points defining
the main features

42. IRDC India
www.irdcindia.com
Presenter: IRDC India
info@irdcindia.com
9 December 2012 42
Contd..
• Apply a Procrustes analysis to align the sets of points ,x, and build a shape
model
• Warp each training s other points match those of the mean shape,obtaining a
“shape-free patch”
• This is a raster scanned into a texture vector, g, whcih is normalized by
applying a linear trasnformation where 1 is vector of
ones , µs and σs
2 are the mean and varaince of elements of g.
• After normalization, gT 1=0 and |g|=1 and calculate texture model by taking
eigen analysis of g.
• Finally the correaltions between shape and texture are learned to generate a
combined model ( as seen in previous paper)
ssgg σµ /)1( −→

43. IRDC India
www.irdcindia.com
Presenter: IRDC India
info@irdcindia.com
9 December 2012 43
Contd..
•To be continued...

44. IRDC India
www.irdcindia.com
Presenter: IRDC India
info@irdcindia.com
9 December 2012 44
Thank you
Queries?
IRDC India
info@irdcindia.com
visit us at www.irdcindia.com