The document discusses feature extraction methods, emphasizing their role in transforming raw data into useful features for machine learning algorithms. It outlines various techniques, such as dimensionality reduction and kernel PCA, while addressing current challenges in the field, including the scalability of extraction methods and the relationship between supervised and unsupervised approaches. Empirical results demonstrate significant accuracy improvements across different applications, affirming the importance of effective feature extraction in enhancing learning outcomes.

1. Feature Extraction: Modern
Questions and Challenges
Dmitry Storcheus
Google Research
co-authors: Mehryar Mohri (NYU Courant), Afshin Rostamizadeh (Google)

2. Feature Extraction (FE)
Def: Feature Extraction (FE) is any algorithm that transformation raw data into features that
can be used as an input for a learning algorithm.
Examples
Construct bag-of-words vector from an email
Remove stopwords in a sentence
Apply PCA projection to high-dimensional data

3. Why Feature Extraction?
Excessive amount of raw features available (image classification,
spam detection)
Learning algorithms are already well-studied
No ML algorithm can perform stable without feature engineering,
but if features are extracted well, even linear methods show great
results
Companies invest in feature extraction pipelines

4. Goals of this talk
Structured survey of Feature Extraction methods
Open research problems + some solutions from Google
Empirical advice

5. Benefits of FE
Increase learning accuracy by taking out the most significant
information from raw data (Guyon, 2003).
Denoising
Reduce overfitting to train set
Memory optimization
Train and inference time optimization

6. Classification of FE methods
Variable subset selection [Blum, 1997; Kohavi, 1997]: Choose best k
variables out of existing ones
Feature Construction: make new variables
Normalization (Lp unit ball or custom metric)
Nonlinear transformation to existing variables
Feature Crossing
Clustering [Duda, 2001]

7. Classification of FE methods
Dimensionality Reduction: project data into low-dimensional subspace
Principal Component Analysis [Pearson, 1901]
Linear Discriminant Analysis [Fisher, 1938]
Random Projection [Hegde, 2008]
Manifold Learning: project data onto a nonlinear low-dimensional manifold
Isometric Feature Mapping [Tenenbaum, 2000]
Locally Linear Embedding [Roweis, 2000]
Laplacian Eigenmap [Belkin, 2003]

8. Classification of FE methods
Distance metric learning: learn a custom distance function on data
Local LDA [Hastie, 1996]
Relevance Component Analysis [Bar-Hillel, 2003]
Multiple Kernel Learning [Cortes, 2009]
Representation Learning [Bengio, 2013]: extracting features at each level of neural
network
Autoencoders [Bengio, 2007]
Restricted Bolzmann Machines [Hinton, 2003]

9. Questions and challenges
Should Feature Extraction be supervised?
Should Feature Extraction be coupled with a classifier?
How to make Feature Extraction methods scalable?
What is the connection between convex and non-convex
methods?

10. Questions and challenges
Should Feature Extraction be supervised?
Should Feature Extraction be coupled with a classifier?
We address these 2 questions on the example of nonlinear
dimensionality reduction

11. Dimensionality reduction
Determine a lower dimensional space preserving various geometric properties of the input.
X PX
P
PCA: captures variance
Isomap: preserves distances along manifold
MVU: preserves angles
()
Kernel PCA
with specific
kernel function
[Ham et al.,2004]

12. Principal Component Analysis
Setting
real-valued training sample of size m
mean centered data matrix
sample covariance matrix
matrix of top d eigenvectors of is
PCA of is
X 2 Rn⇥m
C =
1
m
XX>
U 2 Rm⇥dC
X U>
X

13. Principal Component Analysis
Pseudocode
input
centered data matrix
number of principal components
1. compute
2. = TopEigenVectors // LAPACK:dsyevx(), runtime
return
X
d
C =
1
m
XX>
C, dU O(dm2
)
U>
X

14. Principal Component Analysis
Empirical results of PCA
up to +10% accuracy improvement for face recognition [Yang, 2004]
+3% improvement in accuracy on drug discovery data [Janecek, 2008]
+7% accuracy on email classification with 128 principal components out of 57021 total [Gomez, 2008]
Eigenvalue
0
0.25
0.5
0.75
1
1 2 3 4 5 6 7 8 9 10
Eigenvalue
0
0.25
0.5
0.75
1
1 2 3 4 5 6 7 8 9 10

15. Dimensionality reduction
Determine a lower dimensional space preserving various geometric properties of the input.
X PX
P
PCA: captures variance
Isomap: preserves distances along manifold
MVU: preserves angles
()
Kernel PCA
with specific
kernel function
[Ham et al.,2004]

16. Kernel PCA
Problem set up: doing PCA in Reproducing Kernel Hilbert Space
training sample
Kernel function
example
Reproducing Kernel Hilbert Space and a map
K(xi, xj) : Rn
⇥ Rn
7! R
S = x1, ..., xm 2 Rn
K(xi, xj) = exp
✓
kxi xjk2
2 2
◆
HK K : Rn
7! HK
K(xi, xj) = h K(xi), K(xj)iHK

17. Kernel PCA
Problem set up
is a sample covariance operator of kernel
eigenspace of corresponding to top eigenvalues
is principal component projection
Kernel PCA of point is
C : HK 7! HK K
U 2 HK C d
⇧U : H 7! U 2 H
x ⇧U K(x)

18. Kernel PCA - pseudocode
input
sample
number of principal components
1. compute normalized kernel matrix , s.t.
2. = TopEigenVectors
3. =TopEigenValues
4. for i=1 to m
for k=1 to d
return matrix of coordinates
d
U
K [K]i,j =
1
m
K(xi, xj)
K, d
S = x1, ..., xm 2 Rn
[⇧]k,i =
1
p
k
mX
t=1
K(xi, xt)Ut,k
⇧
K, d

19. Kernel PCA
Empirical results
[Sholkopf, 1998] on USPS dataset (handwritten digits) linear
SVM shows 91% accuracy, while Kernel PCA+ SVM gives 96%
accuracy with only 20% of total principal components

20. Generalized Nonlinear Dimensionality Reduction
Space: let be the reproducing Hilbert space of some kernel
and let be the feature map of kernel
Projection: let be some orthogonal projection in
Dim reduction applied to point isx 2 X
K: x 7! K(x, ·)
P (x)
K
P
HK
HK

21. Open problems
How to choose kernel function K and projection P?
Different kernel means different nonlinear dim reduction method
Can we automatically adjust kernel functor for different
problems?
Can this be done in supervised manner?

22. How to choose P and K?
Traditionally the choice is based on optimizing geometric
properties (e.g. PCA, Isomap, MVU).
However is it a good choice if we use dimensionality reduction
for feature generation? E.g. we want to use reduced data as an
input for a learning problem?

23. Learning on reduced data: a bad example
-3
-2
-1
0
1
2
3
-3 -2 -1 0 1 2 3
u2
u1
C =
✓
4 0
0 1
◆
u1 =
✓
1
0
◆
u2 =
✓
0
1
◆
1 = 4
2 = 1
-3
-2
-1
0
1
2
3
-3 -2 -1 0 1 2 3
u2
u1
Classification error 50%

24. Coupled Nonlinear Dimensionality Reduction
Learner receives:
p PSD kernels
Learner constructs a space , which is the reproducing space of a
mixed kernel
Learner constructs a projection which is the Kernel PCA
projection with kernel
K1, . . . , Kp
H
Kµ =
pX
k=1
µkKk
Kµ
⇧U

25. Hypothesis set
Hypothesis set
Consists of linear maps in the projected subspace of
Parametrized by and
Feature map is the feature mapping of kernel
Projection is based on the unlabeled set
Regularization is a convex set from which is selected.
⇧U : H 7! U 2 H
H
H =
⇢
x 7! hw, ⇧U (x)iH : kwkH  1, µ 2 M .
: X 7! H Kµ
U
M µ
w µ

26. Objective function
For any convex loss function
the coupled training of dimensionality reduction + classifier is
h = argminkwk1;µ2M L(w, µ)
L(w, µ)

27. Supervised Nonlinear Dimensionality Reduction -
conclusion [Storcheus, 2015]
Learning error depends on log(p), take many base kernels
Joint learning of kernel and separation hyperplane is suggested
Automatic learning of dim reduction method by learning a kernel function
Coupled algorithm:
Structural risk minimization
Fit and by minimizing training loss.w µ

28. Thank you for your attention

29. References-1
1. Guyon, Isabelle, and André Elisseeff. "An introduction to variable and feature selection." The Journal of Machine Learning
Research 3 (2003): 1157-1182.
2. Blum,Avrim L., and Pat Langley. "Selection of relevant features and examples in machine learning." Artificial intelligence 97, no.
1 (1997): 245-271.
3. Duda, Richard O., Peter E. Hart, and David G. Stork. "Unsupervised learning and clustering." Pattern classification (2001):
519-598.
4. Hegde, Chinmay, Michael Wakin, and Richard Baraniuk. "Random projections for manifold learning." In Advances in neural
information processing systems, pp. 641-648. 2008.
5. Tenenbaum, Joshua B.,Vin De Silva, and John C. Langford. "A global geometric framework for nonlinear dimensionality
reduction." Science 290, no. 5500 (2000): 2319-2323.
6. Roweis, Sam T., and Lawrence K. Saul. "Nonlinear dimensionality reduction by locally linear embedding." Science 290, no. 5500
(2000): 2323-2326.
7. Belkin, Mikhail, and Partha Niyogi. "Laplacian eigenmaps for dimensionality reduction and data representation." Neural
computation 15, no. 6 (2003): 1373-1396.
8. Cortes, Corinna, Mehryar Mohri, and Afshin Rostamizadeh. "Learning non-linear combinations of kernels." In Advances in

30. References-II
1. Yang, Jian, David Zhang,Alejandro F. Frangi, and Jing-yuYang. "Two-dimensional PCA: a new approach to appearance-based
face representation and recognition." Pattern Analysis and Machine Intelligence, IEEE Transactions on 26, no. 1 (2004):
131-137.
2. Janecek,Andreas GK, and Wilfried N. Gansterer. "A comparison of classiffication accuracy achieved with wrappers, filters
and PCA'." In Workshop on New Challenges for Feature Selection in Data Mining and Knowledge Discovery. 2008.
3. Gomez, Juan Carlos, and Marie-Francine Moens. "PCA document reconstruction for email classification." Computational
Statistics & Data Analysis 56, no. 3 (2012): 741-751.
4. Schölkopf, Bernhard,Alexander Smola, and Klaus-Robert Müller. "Kernel principal component analysis." In Artificial
Neural Networks—ICANN'97, pp. 583-588. Springer Berlin Heidelberg, 1997.
5. Storcheus, Dmitry, Mehryar Mohri, and Afshin Rostamizadeh. "Foundations of Coupled Nonlinear Dimensionality
Reduction." arXiv preprint arXiv:1509.08880 (2015).
6. Bengio,Yoshua,Aaron Courville, and PierreVincent. "Representation learning:A review and new perspectives." Pattern
Analysis and Machine Intelligence, IEEE Transactions on 35, no. 8 (2013): 1798-1828.
7. Ham, Jihun, Daniel D. Lee, Sebastian Mika, and Bernhard Schölkopf. "A kernel view of the dimensionality reduction of
manifolds." In Proceedings of the twenty-first international conference on Machine learning, p. 47.ACM, 2004.

31. Back-up slides

32. General Framework
Define a learning scenario that describes every FE method
Implement this scenario by Kernel PCA algorithm that generalizes most
feature extraction methods
Linear dimensionality reduction, manifold learning, shallow neural nets

33. FE learning scenario
Learner receives:
(partially) labeled training sample of size
Drawn over input space
Feature space
Feature Extraction:
Family of functions
Classification:
Family of functions
m
˜S = ((˜x1, y1), . . . , (˜xm, ym))
˜X
X
F : ˜X 7! X
H : X 7! { 1, 1}

34. FE Learning scenario
FE stage
for any feature extraction function let the features extracted from input data
point be
apply to training set to get
Classification stage
Apply to sample with extracted features
Testing stage:
the classification of a test point is a composition
xi = f( ˜xi)
f 2 F
˜S S = ((x1, y1), ..., (xm, ym))
h 2 H S
x h(f(x))

35. FE Learning problem
The learning problem is to find best and
How?
By minimizing loss function on training set
f?
2 F h?
2 H

36. FE Learning problem
Types of loss functions w.r.t. feature extraction
unsupervised
supervised decoupled
supervised coupled

37. Different loss functions - different methods
for example, if F are orthogonal projections and X star R^d
L is reconstruction error, then f^star is PCA Projection
L is scattered distances, the f^star is multidimensional scaling
Variance within classes - LDA

38. Ongoing work: algorithm

39. Computation of hypothesis
Labeled training sample .
Compute for .
Indexing: index over the base kernels and index over eigenvalues.
Eigenvalues of have the form .
Binary selection variables if eigenspace of is included in
projection.
Coordinates of in the range of are .
Other constants relative to the training set are .
k 2 [1, p] j 2 [1, u]
CU µk j(CU,k)
⇠k,j = 1
⇧Uw 2 H zk,j
h(xn)
S = ((x1, y1), . . . , (xm, ym))
h(xn) xn 2 S
µk j(CU,k)
ck,j(xn)

40. Computation of hypothesis
Numerical expression
optimize for and
Constraints
h(xn)
h(xn) =
X
k,j
⇠k,jck,j(xn)zk,j
p
µk
µ z
kzk2  1
µ 2 M

41. Optimization problem
Minimize a training loss
Over a convex set
kzk2  1
µ 2 M
min
µ,z
1
m
X
n
L
X
k,j
⇠k,jck,j(xn)zk,j
p
µk, yn
!