1. Unsupervised Joint Alignment,
Clustering, and Feature Learning
Marwan A. Mattar
!
PhD Thesis Defense
December 12, 2013

2. Unsupervised Joint Alignment,
Clustering, and Feature Learning

3. Research Goal
Develop an unsupervised data set-agnostic processing module
that includes alignment, clustering, and feature learning

4. Joint Alignment
Transform instances to make data set more coherent

5. Joint Alignment: Domains
• Computer vision: remove spatial variability in images
• Speech: remove temporal variability in speech signals
• Psychology: recover stimulus onset in ERP signals
• Radiology: remove bias in MRI scans
• ...

6. Joint Alignment != Pairwise Alignment
• Pairwise alignment involves two instances
• Joint alignment can be achieved through (N - 1) pairwise alignments
• Can be ineffective (local view & optimization)

7. Joint Alignment: Unsupervised
• Supervision: fiducial points or examples of transforms
• Can be expensive to obtain
• We focus on unsupervised joint alignment

8. Joint Alignment
• Input:
• Data set,
• Transformation function,
• Output:
• Transformed data set, s.t. is maximized
• Transformation parameters,
{x1, x2, . . . , xN }
{y1, y2, . . . , yN }
yi = ⌧(xi, ⇢i)
{⇢1, ⇢2, . . . , ⇢N }
S(y)

9. Joint Alignment: Previous Work
• Procrustes analysis (1939 & 1970’s)
• Congealing framework (2000)
• Domain-specific algorithms (e.g. curves or images)
• Data-set specific algorithms (e.g. ERP)

10. Previous Work: Congealing
(Miller et al. 2000)
• General framework, widely applicable
• Utilizes entropy-based objective function
• Iterative conditional maximization optimization
• Expectation regularization step: E[⇢] = 0

11. Research Goals
• Develop a framework for joint alignment
• Handle complexities due to multi-modalities and representation
!
• Release open-source implementations of all models
clustering feature learning

12. Research Goals
• Develop a
• Handle complexities due to
!
• Release
clustering deep learning
Develop an unsupervised data set-agnostic processing module
that includes alignment, clustering, and feature learning

13. Today ....
alignment
clustering feature learning

14. Today ....
alignment
clustering feature learning

15. Today ....
alignment
clustering feature learning

16. Today ....
alignment
clustering feature learning

17. Future ....
alignment
clustering feature learning

18. Curve Alignment

19. Curve Alignment: Objectives
• Adapt congealing framework to 1-dimensional curves
• Nonparametric: entropy-based objective function
• Efficient: global transformation parametrization
• Experiment with effective parameterizations
• Evaluate effect of alignment on classification

20. Curve Alignment: Objective Function
• Input: curves of length
• Sum of location-wise entropy (or variance)
!
!
• Marginal entropy computed usingVasicek estimator
{yn(t)}N
n=1 ⇠ Y tS(y1, y2, . . . , yN ) =
TX
t=0
H(Y t
) H(Y)
ˆH(x1, x2, . . . , xN ) /
X
log(x(mi+m+1)
x(mi+1)
)
N T

21. Curve Alignment: Optimization
• Conditional maximization with expectation regularization step
• Step sizes are sampled:
• Dampen parameters:
p ⇠ U(0, p)

22. Curve Alignment:Transformation
y(t) = ↵g(t0
)x(t0
) + t0
= h(t)
linear amplitude
scaling
non-linear time
warping
non-linear amplitude
scaling
amplitude
translation

23. Curve Alignment:Transformation
• is a smooth monotone function
• Utilize monotonicity operator of Ramsey (1998)
y(t) = ↵g(t0
)x(t0
) + t0
= h(t)
linear amplitude
scaling
non-linear time
warping
non-linear amplitude
scaling
amplitude
translation
h(t)
unconstrained
function
monotonicity
operator

24. Curve Alignment:Transformation
• Unconstrained and estimated using Fourier basis
• 2 and 7 frequencies, respectively, were sufficient
• Total # of parameters: 20
y(t) = ↵g(t0
)x(t0
) + t0
= h(t)
linear amplitude
scaling
non-linear time
warping
non-linear amplitude
scaling
amplitude
translation
w(t) g(t)

25. Curve Alignment: Experiments
• Alignment of synthetic data sets
• Alignment of real data sets
• Improving classification with unsupervised joint alignment

26. Curve Alignment: Synthetic
• 5 curves x 5 transformations x 3 levels of difficulty = 75 data sets
• ~90% of the data sets converged to correct alignments

27. Curve Alignment: Synthetic
Non-linear time scaling

28. Curve Alignment: Synthetic
All transformations

29. Curve Alignment: Synthetic
Failure case

30. Curve Alignment: Real
2-class data set, color-coded

31. Curve Alignment: Comparison to CPM
TIC data set
Original w/ variance

32. Curve Alignment: Comparison to CPM
TIC data set
Original w/ entropy

33. Curve Alignment: Classification
Aligned
Data Set
Transformation
Parameters
Original
Data

34. Curve Alignment: Classification
• Perform unsupervised alignment
• Represent each instance by its original + aligned + transformation
• Improve the performance of an SVM

35. Curve Alignment: Classification
Method Accuracy
Original Curves (no alignment) 78.83%
Dynamic Time Warping (pairwise alignment) 82.35%
13 Data Sets from UCRTime Series Repository

36. Curve Alignment: Classification
Method Accuracy
Original Curves (no alignment) 78.83%
Dynamic Time Warping (pairwise alignment) 82.35%
Joint Alignment (entropy + non-linear time) 82.97%
13 Data Sets from UCRTime Series Repository

37. Curve Alignment: Classification
Method Accuracy
Original Curves (no alignment) 78.83%
Dynamic Time Warping (pairwise alignment) 82.35%
Joint Alignment (entropy + non-linear time) 82.97%
Joint Alignment (entropy + all transformations) 84.12%
13 Data Sets from UCRTime Series Repository

38. Curve Alignment: Classification
Method Accuracy
Original Curves (no alignment) 78.83%
Dynamic Time Warping (pairwise alignment) 82.35%
Joint Alignment (entropy + non-linear time) 82.97%
Joint Alignment (entropy + all transformations) 84.12%
Joint Alignment (variance + all transformations) 85.46%
13 Data Sets from UCRTime Series Repository

39. Curve Alignment: Classification
Method Accuracy
Original Curves (no alignment) 78.83%
Dynamic Time Warping (pairwise alignment) 82.35%
Joint Alignment (entropy + non-linear time) 82.97%
Joint Alignment (entropy + all transformations) 84.12%
Joint Alignment (variance + all transformations) 85.46%
Joint Alignment (variance + all trans. + MKL) 85.79%
13 Data Sets from UCRTime Series Repository

40. Curve Alignment: Classification
13 Data Sets from UCRTime Series Repository

41. Curve Alignment: Conclusions
• Effective on 1-dimensional curves
• Small # of parameters needed
• Unsupervised alignment improves classification

42. Curve Alignment: Drawbacks
• Lack of a feature representation
• Independence assumption in entropy computation
• Results in co-alignment and difficulty aligning complex data sets
• Transformation regularization is ad hoc

43. Alignment with CRBMs

44. CRBM Alignment: Objectives
• Alignment of complex data sets requires an appropriate representation
• Automate feature selection
• Use statistics of data set to learn an appropriate feature representation
• Experimented with convolutional restricted Boltzmann machines

45. !
• Weights between visible units and hidden unit utilized as feature detector
• Hidden units are binary and activate if feature is detected
• Higher layers represent more complex features
CRBM Alignment: Feature Learning
v1 v2 v3 v4 v5 v6 v7
h1 h2 h3 h4
v1 v2 v3 v4 v5 v6 v7
h1
1 h1
2
h1
3 h1
4
p1
2p1
1
RBM
CRBM
(1 group)

46. CRBM Alignment: Overview
• Train a CRBM using unaligned data
• Use pooling unit activation probability as feature representation
• Use (stochastic) congealing algorithm
• Applied successfully to both curves and face images

47. CRBM Alignment: Curve Features
• Trained a CRBM on training data from 13 UCR data sets
• Learned 32 filters of length 10, with pooling area of 5
• 576 pooling unit activations

48. CRBM Alignment: Curve Features

49. CRBM Alignment: Curve Features

50. CRBM Alignment: Curve Features

51. CRBM Alignment: Classification
Method
Accuracy
Alignment Classification
None
Raw 76.89%
13 Data Sets from UCRTime Series Repository

52. CRBM Alignment: Classification
Method
Accuracy
Alignment Classification
None
Raw 76.89%
Raw Raw 84.63%
13 Data Sets from UCRTime Series Repository

53. CRBM Alignment: Classification
Method
Accuracy
Alignment Classification
None
Raw 76.89%
Raw Raw 84.63%
None
CRBM 84.96%
13 Data Sets from UCRTime Series Repository

54. CRBM Alignment: Classification
Method
Accuracy
Alignment Classification
None
Raw 76.89%
Raw Raw 84.63%
None
CRBM 84.96%
CRBM CRBM 87.40%
13 Data Sets from UCRTime Series Repository

55. CRBM Alignment: Classification
13 Data Sets from UCRTime Series Repository

56. CRBM Alignment: FaceVerification
• Use alignment as pre-processing for face verification task
• Use alternative / consistent feature representation for verification step

57. CRBM Alignment: Filters
CRBM
32 filters, 10x10 size
pooling: 5x5
KMean
clusters

58. CRBM Alignment: Classification
Method Accuracy
Original (no alignment) 74.2%
Alignment with SIFT features 75.8%
Alignment with CRBM features 82.0%
Supervised alignment 80.5%
View 1 from LFW

59. CRBM Alignment: Summary
• Effective for alignment of both curves & faces
• Effective for curve classification

60. Joint Alignment & Clustering

61. JAC: Objectives
• Develop nonparametric/unsupervised alignment & clustering framework
• Applicable to any domain
• Allows use of continuous transformations
• Discovers # clusters automatically
• Regularize transformations principally
• Evaluate model on curve and image data sets

62. JAC: Objectives
• Simultaneous > sequentially
!
!
!
!
• KMeans clustering accuracy = 54% (almost random guessing)

63. JAC: Overview
Bayesian alignment
(assumes single mode)
Infinite Bayesian
alignment & clustering

64. JAC: Bayesian Alignment
xi = ⌧(yi, ⇢i)

65. JAC: Bayesian Alignment
• Explicit regularization of transformations
• Direct maximization that treats transformation function as black box
• Low memory footprint with sufficient stats

66. JAC: Bayesian Alignment
Average pairwise distance: 8.7 (before), 5.2 (with congealing), 5.1 (with BA)

67. JAC: Bayesian Alignment
75 synthetic curve data sets

68. JAC: Bayesian Alignment
Sample result

69. JAC: Model
Bayesian alignment
Single set of parameters
Alignment & Clustering
Potentially infinite parameters

70. JAC: Model
Blocked Gibbs sampler: Direct generalization of BA
transformation termdata termcluster term (CRP)
(zi, ⇢i) ⇠ p(zi | z i; )p(yi | yzi
; )p(⇢i | ⇢zi
; ↵)

71. JAC: Model
Second sampler integrates out transformations
zi ⇠ p(zi | z i; )p(xi | x i, z; , ↵)
Transformation parameter updated with EM

72. JAC: Digits ‘4’ & ‘9’

73. JAC: Digits ‘4’ & ‘9’
Method Clustering Alignment
Original - 4.54
KMeans 54.0%
-Affinity Propagation 82.0% (3)
Infinite mixture model 86.0% (4)

74. JAC: Digits ‘4’ & ‘9’
Method Clustering Alignment
Original - 4.54
KMeans 54.0%
-Affinity Propagation 82.0% (3)
Infinite mixture model 86.0% (4)
Congealing (+KMeans) 90.0% 1.80

75. JAC: Digits ‘4’ & ‘9’
Method Clustering Alignment
Original - 4.54
KMeans 54.0%
-Affinity Propagation 82.0% (3)
Infinite mixture model 86.0% (4)
Congealing (+KMeans) 90.0% 1.80
Unsupervised JAC 94.0% (2) 1.20

76. JAC: Digits ‘4’ & ‘9’

77. JAC:All 10 Digit Classes

78. JAC:All 10 Digit Classes
Method Clustering Alignment
Original - 5.22
KMeans 62.5%
-Affinity Propagation 67.5% (14)
Infinite mixture model 69.0% (13)

79. JAC:All 10 Digit Classes
Method Clustering Alignment
Original - 5.22
KMeans 62.5%
-Affinity Propagation 67.5% (14)
Infinite mixture model 69.0% (13)
Congealing (+KMeans) 64.0% 3.43
Liu et al. 56.5% / 73.7% 3.80

80. JAC:All 10 Digit Classes
Method Clustering Alignment
Original - 5.22
KMeans 62.5%
-Affinity Propagation 67.5% (14)
Infinite mixture model 69.0% (13)
Congealing (+KMeans) 64.0% 3.43
Liu et al. 56.5% / 73.7% 3.80
Unsupervised JAC 87.0% (11) 1.58

81. JAC:All 10 Digit Classes

82. JAC: ECG Heart Data

83. JAC: ECG Heart Data

84. JAC: ECG Heart Data
Method Clustering Alignment
Original - 93.6
KMeans (2 clusters) 58.70%
-
KMeans (5 clusters) 82.61%
Affinity Propagation 65.22% (4)
Infinite mixture model 78.26% (2)

85. JAC: ECG Heart Data
Method Clustering Alignment
Original - 93.6
KMeans (2 clusters) 58.70%
-
KMeans (5 clusters) 82.61%
Affinity Propagation 65.22% (4)
Infinite mixture model 78.26% (2)
Congealing (+KMeans, 5 clusters) 76.09% 53.85

86. JAC: ECG Heart Data
Method Clustering Alignment
Original - 93.6
KMeans (2 clusters) 58.70%
-
KMeans (5 clusters) 82.61%
Affinity Propagation 65.22% (4)
Infinite mixture model 78.26% (2)
Congealing (+KMeans, 5 clusters) 76.09% 53.85
Unsupervised JAC 86.96% (5) 5.93

87. JAC: Summary
• Effective, large improvements over prior work
• Allows finite, semi-supervised, online & distributed variations
• Modular implementation

88. Thesis Summary
Curve Alignment
Alignment &
Feature Learning
Alignment &
Clustering

89. Future Directions

90. Future: Closing the Loop
• Simultaneous unsupervised alignment, clustering, and feature learning
• Update learned features through alignment
• Different filters / weights for each cluster
• Multi-resolution framework using different network depths

91. Research Goal
Develop an unsupervised data set-agnostic processing module
that includes alignment, clustering, and feature learning

92. Thank you.
Photo credit: Gary B. Huang

93. Previous Work: Procrustes Analysis
(Mosier 1939, Hartley & Cattel 1962, Kristof & Wingersky 1971)
• Framework for computing optimal matching under transformations
• Residual sum-of-squares distance called procrustes statistic
• Applied to joint alignment and several other applications
!
!

94. Previous Work: Procrustes Analysis
(Mosier 1939, Hartley & Cattel 1962, Kristof & Wingersky 1971)
• Framework for computing optimal matching under transformations
• Residual sum-of-squares distance called procrustes statistic
• Applied to joint alignment and several other applications
• Drawbacks:
• Mean may not be appropriate reference
• Assumes unimodal data set

95. Previous Work: Procrustes Analysis
(Mosier 1939, Hartley & Cattel 1962, Kristof & Wingersky 1971)

96. Previous Work: Congealing
(Binary images of handwritten digits: Miller et al. 2000)
before after

97. Previous Work: Congealing
(Bias removal in MRI scans: Learned-Miller et al. 2004 & 2005)
before after

98. Previous Work: Congealing
• Complex images of cars and faces
• 3D MRI scans
• Images of Drosophila discs
• Facial contour labeling

99. Previous Work: Congealing
(Grayscale images of faces: Huang et al. 2007 & 2012)

100. Previous Work: Congealing
(3D MRI scans: Zollei et al. 2005)

101. Previous Work: Curves
• HMM-based models: Continuous Profile Model & Segmental HMM
• Many parameters
• Assumes unimodal data set
• Regression-based models:Transformed mixture of regression
• Requires # of clusters
• Only linear transformations

102. Previous Work: Images
• Congealing variants:
• Specific to image domains
• Improved performance on digit and simple face data sets
• Clustering-based models

103. h(t) =
1
Z
Z t
0

exp
✓Z r=t
0
w(s)ds
◆
dr = (w(t))
Curve Alignment:Transformation
• is a smooth monotone function
• Utilize monotonicity operator of Ramsey (1998)
y(t) = ↵g(t0
)x(t0
) + t0
= h(t)
linear amplitude
scaling
non-linear time
warping
non-linear amplitude
scaling
amplitude
translation
h(t)
d2
h
dt2
= w
dh
dt

104. h(t) =
1
Z
Z t
0

exp
✓Z r=t
0
w(s)ds
◆
dr = (w(t))
Curve Alignment:Transformation
• is a smooth monotone function
• Utilize monotonicity operator of Ramsey (1998)
y(t) = ↵g(t0
)x(t0
) + t0
= h(t)
linear amplitude
scaling
non-linear time
warping
non-linear amplitude
scaling
amplitude
translation
h(t)
d2
h
dt2
= w
dh
dt
unconstrained
function
monotonicity
operator

105. Curve Congealing: Drawbacks
Co-alignment
original mean supervised unsupervised

106. Curve Congealing: Drawbacks
Complex multi-modal
50 100 150 200 250
−3
−2
−1
0
1
2
3
4
Original Data Set
50 100 150 200 250
−3
−2
−1
0
1
2
3
4
with Congealing

107. CRBM Alignment: Overview
• Train a convolutional deep belief network using unaligned images
!
!
!
• Use pooling activations as feature representation
• Use (stochastic) congealing algorithm
CRBM
(1-layer)
CDBN
(2-layer)

108. JAC:Toy Example
(2) Congealing (3) Clustering (4) Alignment and Clustering
(1) Original

109. JAC: Bayesian Alignment
8i=1:N ⇢i ⇠ p(yi | y i; )p(⇢i | ⇢ i; ↵)
Collapsed (Rao-Blackwellized) Gibbs Sampler

110. JAC: Bayesian Alignment
8i=1:N ⇢i ⇠ p(yi | y i; )p(⇢i | ⇢ i; ↵)
⇢i = arg max
⇢i
p(yi | y i; )p(⇢i | ⇢ i; ↵)
Collapsed (Rao-Blackwellized) Gibbs Sampler

111. JAC: Bayesian Alignment
8i=1:N ⇢i ⇠ p(yi | y i; )p(⇢i | ⇢ i; ↵)
⇢i = arg max
⇢i
p(yi | y i; )p(⇢i | ⇢ i; ↵)
data term transformation term
Collapsed (Rao-Blackwellized) Gibbs Sampler

112. JAC:Alignment & Clustering
Second sampler integrates out transformations
zi ⇠ p(zi | z i; )p(xi | x i, z; , ↵)
p(xi | x i, z; , ↵) ⇡
P
l wlp(xi | ⇢l, ✓zi ; )
P
l wl
s.t. 8l=1:L ⇢l ⇠ q(⇢) & wl =
p(⇢l | 'zi ; ↵)
q(⇢l)

113. JAC: Implementation
• visDataSet: represents data set (e.g. images, curves)
• visProcessor: handles feature representation (e.g. HOG)
• visTransform: represents transformation function (e.g. affine trans)
• visSufficientStats: represents exponential family members and priors
• visAlignment: base class for alignment algorithms
• visMixtureModelBase: base class for DP-based clustering/alignment

114. JAC: Implementation
visDataSet
visAlignment
visSufficientStatsvisTransforms
visMixtureModel
visProcessor

115. JAC: Implementation
visDataSet
visAlignment
visSufficientStatsvisTransforms
visMixtureModel
visProcessor
visCongealing

116. JAC: Implementation
visDataSet
visAlignment
visSufficientStatsvisTransforms
visMixtureModel
visProcessor
visDPClustering

117. JAC: Implementation
visDataSet
visAlignment
visSufficientStatsvisTransforms
visMixtureModel
visProcessor
visBayesianAlignment

118. JAC: Implementation
visDataSet
visAlignment
visSufficientStatsvisTransforms
visMixtureModel
visProcessor
visAlignmentClustering

119. JAC: Implementation
visDataSet visSufficientStatsvisTransforms visProcessor
visImageTransforms
visCurveTransforms
visRotatePointTransforms
visVectorDataSet
visImageDataSet
visDiscretize
visHOG
visCDBN
visMultinomial
visGaussian

120. Thank you.
Photo credit: Gary B. Huang

121. Thank you.
Photo credit: Gary B. Huang