This document provides an overview of statistical models for shape and appearance, including statistical shape models, combined appearance models, and matching algorithms like active shape models and active appearance models. It discusses how to build statistical shape models from training images by applying procedures like generalised Procrustes analysis to align shapes and then using principal component analysis to model shape variation. It also explains how to build combined models that correlate shape and texture parameters and how active appearance models work by iteratively updating shape and texture parameters to match a new image.

1. 1
Tim Cootes 2014
Statistical Models of Shape and
Appearance
Tim Cootes
Imaging Sciences,
The University of Manchester
http://personalpages.manchester.ac.uk/staff/timothy.f.cootes
(or search for “Tim Cootes”)
Aim
Interpret images using models of appearance
– ‘explain’the image
Fit Model
Model
Parameters
Why is it difficult?
• Wide variation in shape and appearance
Tim Cootes 2013
Overview
• Statistical Shape Models
• Combined Appearance Models
• Matching Algorithms:
– Active Shape Models
– Active Appearance Models
– Constrained Local Models
• Groupwise registration
– Obtain correspondences to build models
Model Building
Example Images
Shape Models
Appearance Models
Statistical
Analysis
Statistical Shape Models
• Statistical model of shape variation

2. 2
Building Shape Models
• Require labelled training images
– landmarks represent correspondences
T
nn ,y,,y,x,(x )11 x
Shape
• Need to model the variability in shape
• What is shape?
– Geometric information that remains when
location, scale and rotational effects removed
(Kendall)
Same Shape Different Shape
Shape
• More generally
– Shape is the geometric information invariant
to a particular class of transformations
• Transformations:
– Euclidean (translation + rotation)
– Similarity (translation+rotation+scaling)
– Affine
Statistical Shape Models
Given a set of shapes:
• Align shapes into common frame
– Procrustes analysis
• Estimate shape distribution p(x)
– Single Gaussian often sufficient
• Approximate with parameterised model
Pbxx 
Aligning Two Shapes
• Procrustes analysis:
– Find transformation which minimises
– Resulting shapes have
• identical CoG
• approximately the same scale and orientation
2
21 |)(| xx T
Aligning a Set of Shapes
• Generalised Procrustes Analysis
– Find the transformations Ti which minimise
– Where
– Under the constraint that
2
|)(| iiT xm 
)(
1
iiT
n
xm 
1|| m

3. 3
Aligning Shapes
• Generalised Procrustes Analysis
– Align each shape to the mean
Unaligned Align with translation Align with similarity
Aligned Shapes
• Need to model the aligned shapes
x
spaceshape
Dimensionality Reduction
• Co-ordinates often correlated
• Nearby points move together
11bpxx 
1b
x
x
1p
Principal Component Analysis
• Compute eigenvectors of covariance, S
• Eigenvectors : main directions
• Eigenvalues : variance along eigenvector
11p
22 p
x
1x
2x
Dimensionality Reduction
• Data lies in subspace of reduced dim.
• However, for some t,
i
i
nnbb ppxx  11
tjbj  if0
t
)isof(Variance jjb 
Building Shape Models
• Given aligned shapes, { }
• Apply PCA
– Compute mean and eigenvectors of covar.
• P – First t eigenvectors of covar. matrix
• b – Shape model parameters
ix


332211 pppx
Pbxx
bbb

4. 4
Face Model Modes
• 100 people, 4 images each
1Varying b 2Varying b 3Varying b 4Varying b
 332211 pppxx bbb
Hand Shape Model
1Varying b 2Varying b 3Varying b
Spine Shape Model Knee shape model
 1b  2b  3b
Xenios Milidonis & ARCOGEN
Prostate Model Mode 1
Danny Allen
Brain structure modes

5. 5
Appearance Models
• Model both shape and texture
• Use all image information
+ =
Building Texture Models
• For each example, extract texture vector
• Normalise vectors
• Build eigen-model
Texture, g
Warp to
mean
shape
ggbPgg 
gg  /)( 1gg 
Face Texture Model
1b12  12 
2b22  22 
3b32  32 
Combined Models
• Shape and texture often correlated
– When smile, shadows change (texture) and
shape changes
• Learning this correlation leads to more
compact (and specific) model
Learning Correlations
sb
gbModel assuming
shape and texture
independent
Model accounting for
correlations between shape
and texture
Learning Correlations
• For each image in set we have best fitting
shape and texture param.s
• Construct new vector,
• Apply PCA (mean + eigenvec.s of covar.)
gs bb ,









g
s
c
b
Wb
b
cQgg
cQxx
g
s

 Varying c changes both
shape and texture

6. 6
Global Face Model
• 400 images from 100 different people
Note: Mixes ID, expression, head pose etc
Face Variation
Gender Change: Ethnic Group:
Face Variation
Age Change:
Model Matching
Active Shape Models
• Match shape model to new image
• Require:
– Statistical shape model
– Model of image structure at each point
Model Point
Model of Profile
Model of Region
Placing model in image
• The model points are defined in a model
co-ordinate frame
• Must apply global transformation,T, to
place in image
Model Frame Image
Pbxx 
)( PbxX  T
),,,;( sYXT ccx

7. 7
ASM Search Overview
• Local optimisation
• Initialise near target
– Search nearby for best match, X’
– Update parameters to match to X’.
)','( ii YX
Local Structure Models
• Need to search for local match for each point
• Often sufficient to search along profile
• Model
– Strongest edge
– Correlation
– Statistical model of profile/patch
– Discriminative model
Profile Models
• Sometimes true point not on strongest
edge
• Model local structure to help locate the
point
Strongest edge True position
Profile Models
• For each point in model
– For each training image
• Sample values along profile
• Normalise
– Build statistical model
• eg Gaussian PDF using eigen-model approach
Or:
• Train classifier to distinguish true/false positions
Searching Along Profiles
• During search we look along a normal for
the best match for each profile
)(xg
x
))(( xp g
)(xg
Form vector from samples
about x
Search algorithm
• Search along profile
• Update global transformation, T, and
parameters, b, to minimise
2
|)(| PbxX T
),( ii YX

8. 8
Update step
• Hard constraints
• Soft constraints
• Can also weight by quality of local match
tp)p(T  bPbxX subject to|)(|Minimise 2
)(log/|)()(|Minimise 2
r
21
)p(T bPbxX 

Multi-Resolution Search
• Train models at each level of pyramid
– Gaussian pyramid with step size 2
– Use same points but different local models
• Start search at coarse resolution
– Refine at finer resolution
Example : Hip Radiograph
11bpxx 
11  33 1  b
ASM Example: Spine
3D Face Model
Mean
1 2
3
Angela Caunce
Trained on approx. 1000 people
Matching 3D ASMs
• Search for each point independently
– Local model depends on current pose
• Estimate shape param.s + projection
Angela Caunce

9. 9
Face Tracking with a 3D Model
Angela Caunce
Active Shape Models
• Advantages
– Fast, simple, accurate
– Efficient to extend to 3D
• Disadvantages
– Only sparse use of image information
– Treat local models as independent
Active Appearance Models
• We have a statistical appearance model
– Trained from sets of examples
• How do we match to new images?
• Use an “Active Appearance Model”
– Efficient iterative matching method
Model Parameters
• Combined model parameters, c
– Shape parameters
– Texture model parameters
• Pose parameters
• Texture transformation 1gg   mim
c
Q
Q
b
b














2
1
g
s
),,,( sYX cc
),,,,,,( sYX cccp 
Key Step in AAM
• Estimate parameter update from current
image sample
Diff. in Ref Frame
pd
pd
Estimating the update step
Linear:
R estimated
– Using Jacobian
– Using Linear Regression
– Using Canonical Correlation Analysis
Non-linear:
– Boosted classifiers
– Boosted regression
)(sp Fd
Rsp d

10. 10
Justification for Jacobian
• Residual difference:
• Measurements in reference frame:
– Model:
– Sample warped to ref:
• Simplest form : Minimise
– Can add robust kernels
)()()( pIpIpr imm 
cQgpI gm  ˆ)(
)(pIim
2
|)(|)( prp E
Prediction using the Jacobian
p
p
r
prppr δ


 )()( d
)()()E( pprpprpp ddd  T
minimizeTo
TT
p
r
p
r
p
r
JR

















1
)(δ pRrp 
p
r
J



j
i
ij
dp
dr
J 
Taylor expansion:
Estimating Jacobian (gradient)
• Need estimate of the Jacobian:
• Estimate each term by small
displacements on the training set
p
r
J



j
i
ij
dp
dr
J 
d
dd
2
)()( 

jiji
ij
pdrpdr
J
Average over all examples
How Good are the Predictions?
• Predicted dx vs. actual dx over test set
Multi-Resolution Predictions
• Predicted dx vs. actual dx over test set
AAM Algorithm
• Initial estimate Im(p)
• Start at coarse resolution
• At each resolution
– Measure residual error, r(p)
– predict correction dp = Rr
– p  p - dp
– repeat to convergence

11. 11
Face Search Face Search
Sub-cortical Structures
Initial Position Converged
Brain search
3D Model-Based Segmentation
Model of femur Using the model to segment the femur
Regression Approach
• Assume functional relationship
s = normalised intensities, or residuals
• Learn function from displaced training
examples
)(sp Fd
}{ ipd )}({ itruei ppss d
Random perturbations Image samples

12. 12
Linear Regression
Rsp d
Scale Rotation Y-trans Shape 1X-trans Shape 2
Phil Tresadern
frameref.inintensityNormaliseds
Rows of R from PC Regression
R from Linear Regression better than that from Jacobian
Non-linear Regression
• Boosted Haar features:
 

n
i iij Hfp 1
))(( sd
featureHaar:)(siH
function1DLearnt:)(xfi
Phil Tresadern
Linear vs Non-linear
– Non-linear methods perform better for poor
initializations but no better close to solution
Phil Tresadern
Jacobian
Linear Regression
Non-linear Regression
AAM Tracking
• Sequence of AAMs with increasing
resolution
• Generic – trained on range of people
• Search each frame in a few ms
Mircea Ionita,
Phil Tresadern
AAM Developments
• Improved optimisation [Matthews &Baker, IJCV2004]
• Improved features [Cootes:CVPR01]
• 2D+3D AAMs [Xiao:CVPR04]
• Canonical Correlation Analysis [Donner:PAMI06]
• Update matrix computation [Saragih:ICPR06]
• Boosted Appearance Models [Liu:CVPR07]
• Non-linear updates [Saragih:ICCV07]
• Fast boosted AAMs [Tresadern:BMVC10]
• Active Orientation Models [Tzimiropoulos:2013]
• Supervised Descent Method [Xiong:CVPR2013]
• Explicit Shape Regression [Cao:CVPR12]
(And lots of others)
Local Model Matching
Find shape and pose parameters to optimise
Shape param.s Pose params

13. 13
Classification vs Regression
Classification:
• Is this the point?
Classification vs Regression
Classification:
• Is this the point?
Regression:
• Where is the point?
Finding Points using Regression Voting
• Train “Random Forest” to predict offsets
Each patch fed into each tree to predict offset and weight
Finding Points using Regression Voting
• Scan regressor across region
• Each patch produces 1 vote per tree
• Accumulate votes in an array
Fully automated femur detection
• Tested on 839 radiographs (ARCOGEN)
Error <0.9mm for 99% of cases
Claudia Lindner
Knee Radiographs
Median 95%ile 99%ile
Mean point-to-curve error:
< 1mm for 99% of 500 images
Fully automatic search results
using AP knee radiographs.
Claudia Lindner, Xenios Milidonis

14. 14
Groupwise Registration
• Accurate correspondences required to
build shape/appearance models
– Slow/hard to manually annotate data
…
…
Tim Cootes 2011
Goal: Fully Automatic System
• Build models from unlabelled images
… Model
Approach
…
Find the correspondences across the set of images
Construct model in an ‘average’ reference frame
Representing of Deformation
• Use piece-wise linear warp field
– Triangular/Tetrahedral mesh
– Simple, compact, efficient
– Trivially invertible
Groupwise Algorithm
• Initialisation: Affine registration
• Generate control points
• Repeat
– Build model from current data
– For each image
• Optimise positions of control points
• Until happy/dead/conference deadline etc
Image Features
Linear Normalisation
),max(
ˆ
mini
ii
i
gg
g


Local z-normalisation
Raw Image

gg
g i
i
ˆ

1) z-norm
2) Smoothed |Gx|
3) Smoothed |Gy|

15. 15
Registering Faces
• 293 individuals from XM2VTS
• Evolution of robust estimate of mean:
Example Modes
Varying appearance model parameters
(Note shadowing caused by glasses)
Modes of model built from images without glasses:
Hand Radiographs
Registered Mean
Zhang Pei
3D Registration
• Registering 270 3D MR Brain Images
• Evolution of the model mean:
Summary
• Statistical shape models
– Powerful representations of objects
– Usually only need small number of params
• Model matching methods
– ASM/CLM: Local search + Shape Regularisation
– AAMs: Direct prediction of update steps
• Groupwise Registration
– Automatically build models from sets of
images