This document provides an overview of multivariate statistics techniques, including factor analysis, multidimensional scaling, and cluster analysis. [1] Factor analysis aims to reduce the number of variables through principal component analysis or explanatory factor analysis. [2] Multidimensional scaling maps objects into a k-dimensional space to approximate given distance matrices between objects. [3] Cluster analysis groups similar objects into clusters using partitioning or hierarchical methods.

1. Multivariate statistics
Overview
Hoksan | January 10 ,2013 | Rotterdam
Factual
decision
making

2. Why multivariate statistics?
• Observing the correlation or factors between variables
• Summarizing (redundant) variables/observations
• Reduce overfitting issues
Factual
decision
making 2

3. Outline
• Factor analysis:
• Principal Component Analysis
• Explanatory Factor Analysis
• Multidimensional scaling:
• Principle Coordinates Analysis
• Stress Minimization
• Cluster Analysis
Factual
decision
making 3

4. Factor Analysis – idea
• Given 𝑿 (n by m matrix)
• Find p factor loadings 𝒁 from original data 𝑿 with :
• 𝒙 𝒌 ≈ 𝑢1𝑘 𝒛 𝟏 + 𝑢2𝑘 𝒛 𝟐 + ⋯ + 𝑢 𝑗𝑗 𝒛 𝒑 → 𝑿 ≈ 𝒁𝒁
Variance matrix of 𝐙 equals diagonal matrix
• Loadings are mutually uncorrelated
•
Factual
decision
making 4

5. Specific matrix properties
- Normalized data Xs:
- 𝑋 𝑠𝑇 𝑋 𝑠 = correlation matrix:
- Zero mean and equal standard deviation
- 𝑐𝑐𝑐𝑐 𝑋 𝑖 , 𝑋 𝑗 =
∑ 𝑘 𝑥 𝑖𝑖 −𝑥̅ 𝑖 𝑥 𝑗𝑘 −𝑥̅ 𝑗
𝑠𝑠 𝑋 𝑖 ×𝑠𝑠 𝑋 𝑗
• 𝑈 𝑇 𝑈 = identity matrix
• Orthonormal matrix U (rotation matrix)
• If 𝑥 = 𝑎1 𝑧1 + 𝑎2 𝑧2 + 𝑎3 𝑧3 (with 𝑧 𝑖 independent
• Variance of linear combination of uncorrelated variables:
• 𝑥 𝑇 𝑥 = 𝑎1 (𝑧1𝑇 𝑧1 ) + 𝑎2 (𝑧2𝑇 𝑧2 ) + 𝑎3 (𝑧3𝑇 𝑧3 )
2 2 2
variables)
Factual
decision
making 5

6. Factor Analysis – Principal Component Analysis
• Find uncorrelated set of components 𝒁
• Such that the variances 𝑣𝑣𝑣(𝒛 𝒊 ) are maximized
Based on Singular Value Decomposition: 𝐗 = 𝒁 𝒔 𝑫𝑼 𝑻
1 ⋯ 0 1 ⋯ 0
•
𝑍 𝑠𝑇 𝑍 𝑠 = ⋮ ⋱ ⋮ = 𝐼, 𝑈 𝑈= ⋮ ⋱ ⋮ = 𝐼
𝑇
0 ⋯ 1 0 ⋯ 1
–
𝑑1 ⋯ 0
𝐷= ⋮ ⋱ ⋮ = diagional matrix
0 ⋯ 𝑑𝑘
–
Factual
decision
making 6

7. Factor Loadings
Given approximation of X with factors/components:
𝑿 = 𝒁 𝒔 𝑫𝑼 𝑻 (with components 𝒁 𝒔 )
The variances of X equals:
𝒙𝒌 = 𝑢1𝑘 𝑑1 𝒛 𝟏 + 𝑢2𝑘 𝑑2 𝒛 𝟐 + ⋯ + 𝑢 𝑗𝑗 𝑑 𝑝 𝒛 𝒑
𝑣𝑣𝑣 𝒙 𝒌 = 𝑑1 𝑢1𝑘 + 𝑑2 𝑢2𝑘 + ⋯ + 𝑑 2 𝑢2
2 2 2 2
𝑝 𝑗𝑗
� 𝑣𝑣𝑣 𝒙 𝒌 = 𝑑1 + 𝑑2 + ⋯ + 𝑑 2
2 2
𝑝
𝒌 Factual
decision
making 7

8. Factor Analysis – Explanatory Factor Analysis
Find uncorrelated set of factors 𝚵 :
• Similar to PCA: uncorrelated set of factors/components
•
• 𝑿 = 𝚵𝚲 𝑻 + 𝚫, for example:
𝚫 𝑻 𝚫 = diagonal matrix
• Such that the unexplained part Δ is also uncorrelated:
𝚵 𝑻 𝚵 = identity matrix
•
•
Factual
decision
making 8

9. Factor Analysis – Examples
• Decompose correlation matrix 𝐑 into 𝐑 = 𝐃 𝐓 𝐃
Note: Correlation matrix as input is also possible
𝐑 𝑿 = 𝒁 𝒔 𝑫𝑼 𝑻
• In case of PCA:
𝑿 𝑻 𝑿 = 𝑼𝑼𝒁 𝒔𝑻 𝒁 𝒔 𝑫𝑼 𝑻 = 𝑼𝑫 𝟐 𝑼 𝑻
•
•
• Equals eigen decomposition
• Only the component loadings can be calculated, not the
compents itself
Factual
decision
making 9

10. Multidimensional Scaling – idea
• Given distance matrix 𝚫 (n by n matrix)
With coordinates 𝑿 (n by k matrix)
• Map the objects into k-dimensional space
•
Approximating given distance matrix:
2 1/2
𝛿 𝑖𝑖 ≈ 𝑑 𝑖𝑖 = ∑ 𝑥 𝑖𝑖 − 𝑥 𝑗𝑗
•
𝑘
• 𝑎=1
Factual
decision
making 10

11. MDS – Principle Coordinates Analysis
Create full coordinates 𝑿 (n by n-1 matrix) which result in
• Similar to Principal Component Analyis
•
distance matrix
• Perform principal component analysis to get the most of the
variances
• Main differences:
• MDS focuses on the differences/similarities between objects
• FA focuses on the underlying factors/components
Factual
decision
making 11

12. MDS – Stress Minimization
• Find representative coordinates 𝑿 that has approximately distance
Similar to Principle Coordinates Analysis:
matrix equal to 𝚫
But by minimizing the stress value:
𝐦𝐦𝐦 𝝈 𝑿 = � 𝑑 𝑖𝑖 𝑋 − 𝜹 𝑖𝑖
2
𝑖<𝑗≤𝑛
Factual
decision
making 12

13. Cluster Analysis – idea
• Grouping similar objects in clusters
• Two kinds of clustering methods:
• Partitioning methods (k-means)
• Hierarchical methods (dendrogram)
Factual
decision
making 13

14. Questions
? Factual
decision
making 14