Spectral factorization

Spectral Decomposition and its Applications
Senthil Kumarasamy
Gyan Data Private Limited
Senthil Kumarasamy Spectral Decomposition and its Applications Gyan Data Private Limited 1 / 23

Contents
1 Eigen values and Eigen vectors
2 Spectral decomposition
3 Gram Schmidt orthogonalization
4 QR factorization
5 QR algorithm for finding Eigen values
6 Gaussian elimination to find Eigen vectors
7 Implementation in Matlab
8 Applications
Decoupling of linear differential equations
Dimensionality reduction
Filtering
Identification of linear model from data

Eigen values and Eigen vectors
Given a matrix A ∈ Rn×n
Ax = λx
where x ∈ Rn and λ ∈ R. λ is called Eigen value of A, and x is
called the Eigen vector of A corresponding to λ
To ﬁnd Eigen values,
Ax = λx
Ax − λx = 0
(A − λI)x = 0
In order to have non trival solution to the above equation, matrix
A − λI must be singular. Hence |A − λI| = 0
|A − λI| = 0 is called characteristic equation of A. This is a
polynomial of degree n in λ, and it has n solutions namely λ1 . . . λn
Corresponding to each λi, there is an Eigen vector xi

Spectral Decomposition
Ax1 = λ1x1
...
Axn = λnxn
A


| |
x1 . . . xn
| |

 =


| |
x1 . . . xn
| |





λ1
...
λn



A =


| |
x1 . . . xn
| |


P



λ1
...
λn



Σ


| |
x1 . . . xn
| |


−1
P−1
A = PΣP−1

Examples
Find eigen values of A =


0 −1 −3
2 3 3
−2 1 1


Solution
|A − λI| =
0 − λ −1 −3
2 3 − λ 3
−2 1 1 − λ
= −λ
3 − λ 3
1 1 − λ
+ 1
2 3
−2 1 − λ
− 3
2 3 − λ
−2 1
= λ3
− 4λ2
− 4λ + 16
Solving λ3 − 4λ2 − 4λ + 16 = 0, we get λ = 2, −2, 4

Examples contd.


3 2 1
0 2 3
0 0 5



Examples contd.


3 2 1
0 2 3
0 0 5


Solution
|A − λI| =
3 − λ 2 1
0 2 − λ 3
0 0 5 − λ
= (3 − λ)
2 − λ 3
0 5 − λ
− 2
0 3
0 5 − λ
+ 1
0 2 − λ
0 0
= (3 − λ)(2 − λ)(5 − λ)
Solving (3 − λ)(2 − λ)(5 − λ) = 0, we get λ = 3, 2, 5

Examples contd.


3 2 1
0 2 3
0 0 5


Solution
|A − λI| =
3 − λ 2 1
0 2 − λ 3
0 0 5 − λ
= (3 − λ)
2 − λ 3
0 5 − λ
− 2
0 3
0 5 − λ
+ 1
0 2 − λ
0 0
= (3 − λ)(2 − λ)(5 − λ)
Solving (3 − λ)(2 − λ)(5 − λ) = 0, we get λ = 3, 2, 5
Observation: If A is a triangular (Upper or Lower or Diagonal)
matrix, then Eigen values of A are its diagonal elements.

Introduction to QR algorithm
QR algorithm applies Similarity transformations iteratively to
reduce the given matrix to a diagonal matrix
Similarity transformations are those that preserve Eigen values of
the matrix
To get a similarity transformation, QR factorization is employed.
QR factorization uses Gram Schmidt orthogonalization

Gram Schmidt orthogonalization
A =
3 2
1 2
v1 =
3
1
, v2 =
2
2
u1 = v1
e1 = u1
u1
=
3 2
T
√
32+12
= 1√
10
3
1
u2 = v2− (vT
2 e1)e1
Proj of v2 on e1
= 1
5
−2
6
e2 = u2
u2
= 1√
10
−1
3
Q = 1√
10
3 −1
1 3

QR factorization
A = QR
A, Q, R are n × n matrices
QT Q = I
R is upper triangular matrix
Example

0 1 1
1 1 2
0 0 3


A
=


0 1 0
1 0 0
0 0 1


Q


1 1 2
0 1 1
0 0 3


R
QT Q =


0 1 0
1 0 0
0 0 1


T 

0 1 0
1 0 0
0 0 1

 =


1 0 0
0 1 0
0 0 1



QR factorization
A = v1 v2 v3 where v1, v2, v3 are columns of matrix A
Apply Gram Schmidt Orthogonalization to A matrix
u1 = v1
e1 = v1
v1
u2 = v2 − (vT
2 e1)e1
e2 = u2
u2
u3 = v3 − (vT
3 e1)e1 − (vT
3 e2)e2
e3 = u3
u3
QR factorization
A = e1 e2 e3
Q


v1 vT
2 e1 vT
3 e1
0 v2 vT
2 e1
0 0 v3


R

Matlab code for QR factorization
function [ Q , R ] = qr_factorization1(A)
[ r , c ] = size (A) ;
Q = zeros (r , r) ;
R = zeros (r , r) ;
R(1 ,1) = sqrt (sum(A ( : , 1 ) . * A ( : , 1 ) ) ) ;
i f abs (R(1 ,1) ) > 1e−5
Q ( : , 1 ) = A ( : , 1 ) / R(1 ,1) ;
else
Q ( : , 1 ) = 0;
end
f o r i = 2:r
Q ( : , i) = A ( : , i) ;
f o r j = 1:i−1
R(j , i) = sum(A ( : , i) . * Q ( : , j) ) ;
Q ( : , i) = Q ( : , i) − R(j , i) * Q ( : , j) ;
end
R(i , i) = sqrt (sum(Q ( : , i) . * Q ( : , i) ) ) ;
i f abs (R(i , i) ) > 1e−5
Q ( : , i) = Q ( : , i) / R(i , i) ;
else
Q ( : , i) = 0;
end
end
end

QR Algorithm
Algorithm
1 Set Ai = A, the given matrix
2 Q, R = qr factorization(Ai)
3 Set Ai = RQ
4 Perform steps 2 and 3 until convergence
RQ transformation
A = QR
QT A = QT QR = R
RQ = QT
AQ
Similarity transform
Properties of similarity transform
1 Preserve the Eigen values (λ) of A
2 Kill the off diagonal elements of A
Preserves λ
|QT AQ − λI| =
|QT AQ − λQT IQ| =
|QT (A − λI)Q| =
|QT | × |A − λI| × |Q| =
|Q−1| × |A − λI| × |Q| =
|A−λI|×|Q|
|Q| =
|A − λI|

Matlab code for ﬁnding Eigen values
function eig = eigen_values(A)
M = A ;
f o r i = 1:60
[ Q , R ] = qr_factorization(M) ;
M = R*Q ;
end
eig = diag (M) ;
[ eig1 , ind ] = sort ( abs ( eig ) , ' descend ' ) ;
eig = eig (ind) ;
end

Eigen Vectors
(A − λI)
¯A
x = 0
¯Ax = 0
¯Q ¯Rx = 0
¯QT ¯Q ¯Rx = 0
¯Rx = 0
Example:
A =


0 1 1
1 1 2
0 0 3


λ =
−0.62, 1.62, 3.0
For λ = 3

0 1 1
1 1 2
0 0 3

 − 3


1 0 0
0 1 0
0 0 1

 x = 0


−3 1 1
1 −2 2
0 0 0

 x = 0


−0.949 −0.316 0
0.316 −0.949 0
0 0 0




3.162 −1.581 −0.316
0 1.581 −2.214
0 0 0

 x = 0
pre-multiplying both sides by QT , we get


3.162 −1.581 −0.316
0 1.581 −2.214
0 0 0

 x = 0
Slove for x by back substitution

Matlab code for Back Substitution
function x = back_substitution(R)
[ r , c ] = size (R) ;
x = zeros (r) ;
f o r i = r:−1:1
i f abs (R(i , i) ) < 1e−5
x(i) = 1;
continue ;
end
summ = 0.0;
f o r j = r: −1:i
summ = summ + R(i , j) * x(j) ;
end
x(i) = −summ / R(i , i) ;
end
end

Matlab code for ﬁnding Eigen vectors
function [ vec , val ] = eigen_val_vec(M)
[ r , c ] = size (M) ;
val = eigen_values(M) ;
vec = zeros (r) ;
f o r i = 1:r
A = M − eye (r) * val(i) ;
[ Q , R ] = qr_factorization(A) ;
x = back_substitution(R) ;
vec ( : , i) = x / sqrt (sum(x . * x) ) ;
end
end

Applications
Decoupling of differential equations
Consider systems of equations
dx1
dt
= 2x1 + 3x2
dx2
dt
= 2x1 + x2
˙x1
˙x2
˙X
=
2 3
2 1
A
x1
x2
X
˙X = AX
˙X = PΣP−1X
substituting X = PY
P ˙Y = PΣP−1PY
P ˙Y = PΣY
pre-multiplying both side with P−1
P−1
P ˙Y = P−1PΣY
˙Y = ΣY
˙y1
˙y2
=
4 0
0 −1
y1
y2

Applications
Data compression, ﬁltering, model identiﬁcation (PCA)




1 −1 −1 0 0 0
0 1 0 −1 0 0
0 1 0 0 −1 0
0 0 0 1 1 −1












x1
x2
x3
x4
x5
x6








=




0
0
0
0




Data

Applications
Steps to data compression, filtering, model identification (PCA)
Let Z be data matrix. Each column of Z represents a flow variable
Compute S = 1
N ZT Z where N is no. of data points
Find Eigen values, and Eigen vectors of S
Let columns of U1 consists of Eigen vectos corresponding
significant Eigen values
Let columns of U2 consists of Eigen vectors corresponding to
insignificant Eigen values.
Then UT
2 represents the underlying model equations
Compressed data is C = ZU1
Filtered data is F = CUT
1

(Applications)
Matlab code for compression, ﬁltering, and model identiﬁcation
clear a l l ;
clc ;
M = csvread ( ” flow_with_noise . csv ” ) ;
[ r , c ] = size (M) ;
S = M ' * M / r ;
[ eig_vec , eig_val ] = eigen_val_vec(S) ;
basis = eig_vec ( : , 5 : 6 ) ;
model = eig_vec ( : , 1 : 4 ) ' ;
compressed_data = M * basis ;
filtered_data = compressed_data * basis ' ;

Exercises
Limitations of our code
Extend QR factorization code for rectangular matrices
Modify QR Algorithm code for ﬁnding out complex Eigen values
Back substitution code can be extended for handling repeated
Eigen values

Thank you

Spectral factorization

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Spectral factorization

Similar to Spectral factorization (20)

Recently uploaded

Recently uploaded (20)

Spectral factorization