Matrices ii

1
Matrices II
SOLO HERMELIN
Updated: 20.07.07http://www.solohermelin.com

2
SOLO Matrices
Table of Content
Singular Values
Definitions
Domain and Codomain of a Matrix A
Properties of Square Orthogonal Matrices
Definition of the Singular Values
Geometric Interpretation of Singular Values
Properties of Singular Values
Moore-Penrose Pseudoinverse Matrix Anxm
†
of Amxn .
Householder Transformation
Projection of a vector on a vector .b

a

111
min&min
1
nxmxnxmxn
x
xbxAd
nx

 −=
Computation of Moore-Penrose Pseudoinverse Matrix, A †
Properties of Moore-Penrose Pseudoinverse Matrix, A †
Description of Projections Related to Moore-Penrose Pseudoinverse
Particular case (1) r = n ≤ m:
Particular case (2) r = m ≤ n:

3
SOLO Matrices
Table of Content (continue – 1)
General Solution of Amxn Xnxp = Bmxp
Particular case (1) r = m ≤ n
Particular case (2) r = n ≤ n
General Solution of YpxmAmxn = Cpxn
Particular case (1) r = m ≤ n
Particular case (2) r = n ≤ n
Inverse of Partitioned Matrices
Matrix Inverse Lemmas Identities
Matrix Schwarz Inequality
Trace of a Square Matrix
References

4
SOLO Matrices
Singular Values
Definitions:
Any complex matrix A with n rows (r1, r2,…,rn) and m columns (c1,c2,…,cm)
[ ]m
n
nxm ccc
r
r
r
A ,,, 21
2
1


=














=
can be considered as a linear function (or mapping or transformation) for a
m-dimensional domain to a n-dimensional codomain.
( ) ( ){ }AcodomyAdomxxAyA nxmxnxm ∈⇒∈= 11;:
In the same way its conjugate transpose:
[ ]H
n
HH
H
m
H
H
H
mxn
rrr
c
c
c
A ,,, 21
2
1


=














=
is a linear function (or mapping or transformation) for an-dimensional codomain to
a m-dimensional domain.
( ) ( ){ }AcdomxAcodomyyAxA mxnx
HH
mxn ∈⇒∈= 111111 ;:
Table of Contents

5
SOLO Matrices
Domain and Codomain of a Matrix A
The domain of A can be decomposed into orthogonal subspaces:
( ) ( ) ( )ANARAdom H
⊥
⊕= ( )H
AR
( )AN
( )H
AN
( )AR
xAy =
11 yAx H
=
( )Adomxmx
∈1
11mxx
( )Acodomy nx
∈11
1nxyR (AH
) – is the row space of AH
(dimension r)
N (A) – is the null-space of A (x∈ N (A) ⇔ A x = 0)
or the kernel of A (ker (A)) (dimension m-r)
The codomain of A (domain of AH
) can be decomposed into orthogonal subspaces:
( ) ( ) ( )H
ANARAcodom
⊥
⊕=
R (A) – is the column space of A (dimension r)
N (AH
) – is the null-space of AH
(dimension n-r)
Singular Values
Table of Contents

6
SOLO
Hermitian = Symmetric if A has real components
Hermitian Matrix: AH
= A, Symmetric Matrix: AT
= A
Matrices
Properties of Square Orthogonal Matrices
Use Pease, “Methods of Matrix Algebra”, Mathematics in Science and Engineering
Vol.16, Academic Press 1965
Definitions:
Adjoint Operation (H):
AH
= (A*)T
(* is complex conjugate and T is transpose of the matrix)
Skew-Hermitian = Anti-Symmetric if A has real components.
Skew-Hermitian: AH
= -A, Anti-Symmetric Matrix: AT
=-A
Unitary Matrix: UH
= U-1,
Orthonormal Matix: OT
= O-1
Unitary = Orthonormal if A has real components.
Charles Hermite
1822 - 1901

7
SOLO Matrices
Properties of Square Orthogonal Matrices (continue – 1)
Lemma1:
All the eigenvalues of a hermitian matrix H are real and the eigenvectors are orthogonal.
Proof of Lemma1:
Pre-multiply by :iii xHx λ=
H
ix i
H
iii
H
i xxHxx λ=
and take the conjugate transpose: ( ) i
H
iii
HH
i
H
i
H
i xxxHxHxx
*
λ==
This proves that the eigenvalues of H are real.
Subtract those two equations: ( ) 00
**
≠=→=− i
H
iiii
H
iii xxsincexx λλλλ
From ( ) H
ij
H
jj
H
j
HH
j
H
j xxHxHxHx λλ ====
*
Pre-multiply by and post-multiply by and subtractiii xHx λ=
H
jx H
ij
H
j xHx λ= ix
( ) 0=−→




=
=
i
H
jji
i
H
jji
H
j
i
H
jii
H
j
xx
xxHxx
xxHxx
λλ
λ
λ
0=→≠ i
H
jji xxλλIf
If we can use the Gram-Schmidt procedure to obtain an eigenvector
orthogonal to .
ji λλ = jx~
ix ( )
( ) i
i
H
i
j
H
i
jj x
xx
xx
xx −=~

8
SOLO Matrices
Lemma1:
All the eigenvalues of a hermitian matrix H are real and the eigenvectors are orthogonal.
Proof of Lemma1 (continue – 1):
( ) 0=−→




=
=
i
H
jji
i
H
jji
H
j
i
H
jii
H
j
xx
xxHxx
xxHxx
λλ
λ
λ
0=→≠ i
H
jji xxλλIf
If we can use the Gram-Schmidt procedure to obtain an eigenvector
orthogonal to .
ji λλ = jx~
ix
( )
( ) i
i
H
i
j
H
i
jj x
xx
xx
xx −=~
we can see that
( )
( ) 0~ =−= i
H
i
i
H
i
j
H
i
j
H
ij
H
i xx
xx
xx
xxxx
( )
( )
( )
( )
( )
( ) jii
i
H
i
j
H
i
jiii
i
H
i
j
H
i
jji
i
H
i
j
H
i
jj
xx
xx
xx
xx
xx
xx
xxH
xx
xx
xHxH ~~ λλλλ =








−=−=−=
q.e.d.

9
SOLO Matrices
Lemma2:
Any hermitian matrix H can be factored in H = U Λ UH
where Λ=diag (λ1,λ2,…,λn) and U is unitary i.e. U UH
= UH
U = In.
Proof of Lemma2:
Let normalize the orthogonal eigenvectors of H ;i.e. iii
xxu /:=
or H U = U Λ where U = [u1,u2,…,un]
Because U is a square matrix having orthonormal columns, and is a square matrix,
U is also a unitary matrix satisfying UH
U=U UH
=In. q.e.d.
[ ] [ ]












=
000
00
00
,,,,,, 2
1
2121





λ
λ
nn uuuuuuH

10
SOLO Matrices
Lemma3:
A AH
and AH
A are hermitian matrices that have the same nonzero real positive
eigenvalues.
Proof of Lemma3:
q.e.d.
( ) ( ) HHHHHH
AAAAAA == i.e. A AH
is hermitian, therefore the eigenvalues λi (A AH
)
are real and positive according to Lemma 1.
Suppose ui is a normalized eigenvector of λi (A AH
) ≠0
( ) i
H
ii
H
uAAuAA λ=
Pre-multiply by AH
and define
( )
i
H
H
i
i uA
AA
v
λ
1
:=
( ) ( )
( )
( )
( )
( ) ( ) i
H
ii
H
H
i
i
H
H
i
H
i
i
H
H
i
HH
ii
HH
vAAvAA
AA
uA
AA
AA
uA
AAuAAAuAAA
λ
λ
λ
λ
λ
=→
→=→=
we get
We can see that νi is the eigenvector of AH
A and λi (A AH
) is the corresponding
eigenvalue, meaning that both AH
A and A AH
have the same nonzero eigenvalues.
From ( ) ( ) ( ) ( ) ( ) ( ) 02
2
>==→=→=
i
i
i
H
i
i
H
iH
ii
H
i
H
ii
HH
ii
H
ii
H
v
vA
vv
vAvA
AAvvAAvAAvvAAvAA λλλ
Therefore we can define ( ) 0: >= H
ii
AAλσ

11
SOLO Matrices
Lemma4:
If U is a unitary matrix then all its eigenvalues have unit modulus. .
Proof of Lemma4:
form the inner product
IUUUU HH
==
Consider the set of eigenvalues x1, x2, …, xn which we know to be complete and
iii xUx λ=
nixxIxxUxUxxx iiii
H
ii
H
ii
HH
ii
H
iii ,11
**
=∀==→=== λλλλλ
q.e.d.

12
SOLO Matrices
Proof of Lemma5:
Lemma5:
Every unitary matrix U can be expressed as an exponential matrix:
where H is hermitian (jH is skew-hermitian)
jH
eU =
Since the eigenvalues of U have unit modulus; i.e. we can writenii ,11 =∀=λ
niej Hij
HiHii ,1sincos =∀=+= λ
λλλ
( ){ } jHjjj
eSeeediagSU HnHH
== −1
,,, 21 λλλ

( ){ } 1
21 ,,, −
= SdiagSH HnHH λλλ where:
q.e.d.

13
SOLO Matrices
Table of Contents
Decomposition of Square Matrices:
( ) ( ) ( ) ( )



−
−
++=−++= HHHH
AA
j
jAAAAAAA
22
1
2
1
2
1
( ) ( )H
H
H
AAAA +=



+
2
1
2
1
( ) ( ) ( )HH
H
H
AAAAAA −−=−=





−
2
1
2
1
2
1
here: Hermitian
Skew-Hermitian
( ) ( ) ( )HH
H
H
AA
j
AA
j
AA
j
−
−
=−=



−
−
222
Hermitian
( ) ( )





−
−
++= HH
AA
j
jAAA
22
1 the matrix generalization of the decomposition of
a complex number in the real and imaginary part.

14
SOLO Matrices
Lemma6:
(6.1) Every complex nxm matrix of rank can be factored into:
( )
[ ] ( )
( ) ( ) ( ) ( )















Σ
=Σ=
−
−
−−−
−
H
H
rmxrnxrrn
rmrxH
mxmnxmnxnnxm
xmrm
rxmrxr
rnnxnxr
V
V
UUVUA
2
11
21
00
0



where ( ) 0,,, 21211 >≥≥≥=Σ rrdiagrxr
σσσσσσ 
Unxn and Vmxm are unitary matrices, i.e.:
[ ] [ ]
( ) ( )






=








==








=
−− rnxrn
rxr
H
H
H
H
H
H
I
I
UU
U
U
UU
U
U
UUUU
0
0
21
2
1
2
1
21
[ ] [ ]
( ) ( )






=








==








=
−− rmxrm
rxr
H
H
H
H
H
H
I
I
VV
V
V
VV
V
V
VVVV
0
0
21
2
1
2
1
21
(6.2) σi i=1,…,r are the positive square roots of the nonzero eigenvalues of AH
A or
A AH
and are called the singular values of A.
(6.3) The dyadic expansion of A is: where ui and vi are the columns of U1
and V1 respectively.
∑=
=
r
i
H
iii vuA
1
σ
Singular Values

15
SOLO Matrices
Lemma6 (continue – 1):
(6.5) The columns of V are orthonormal eigenvectors of AH
A: [ ] [ ] ( )
( ) ( ) ( ) 






Σ
=
−−−
−
rmxrmxrrm
rmrxH rxr
VVVVAA
00
0
2
1
2121



i.e. the columns of V1 are the eigenvectors of the nonzero eigenvalues, and the
columns of V2 are the eigenvectors of the zero eigenvalues of AH
A.
(6.6) The following relations exist between U1 and V1
1
111
1
111
−
−
Σ=
Σ=
rxrnxrmxr
rxrmxrnxr
UAV
VAU
H
nxm
nxm
i.e. the columns of U1 are the eigenvectors of the nonzero eigenvalues, and the
columns of U2 are the eigenvectors of the zero eigenvalues of A AH
.
(6.4) The columns of U are orthonormal eigenvectors of A AH
: [ ] [ ] ( )
( ) ( ) ( ) 






Σ
=
−−−
−
rmxrnxrrn
rmrxH rxr
UUUUAA
00
0
2
1
2121



Singular Values

16
SOLO Matrices
( )H
AR
( )AN
( )H
AN
( )AR
xAy =
11
yAx H
=
( )Adomxmx ∈1
11mx
x
( )Acodomy nx
∈11
1nxy
(6.7) The columns U1 of form an orthonormal basis for
the column space of A: ( ) ( )ARUR =1
The columns of U2 form an orthonormal basis for
the nullspace of AH
: ( ) ( ) ( )HH
AANUR ker2 ==
The columns of V1 form an orthonormal basis for the column space of AH
:
( ) ( )H
ARVR =1
The columns of V2 form an orthonormal basis for the nullspace of A:
( ) ( ) ( )AANVR ker2 ==
Singular Values

17
SOLO Matrices
Proof of Lemma 6:
and
From Lemma 3 we have ( )
[ ] ( )
( ) ( ) ( ) ( ) 














 Σ
=Λ=
−
−
−−−
−
H
H
rnxrnxrrn
rnrxH
nxnnxn
H
mxnnxm
xnrn
rxnrxr
rnnxnxrnxn
U
U
UUUUAA
2
1
2
1
211
00
0
( )
[ ] ( )
( ) ( ) ( ) ( ) 














 Σ
=Λ=
−
−
−−−
−
H
H
rmxrmxrrm
rmrxH
mxmmxmnxm
H
mxn
xmrm
rxmrxr
rmmxmxrmxm
V
V
VVVVAA
2
1
2
1
212
00
0
where ( ) 0,,, 21211 >≥≥≥=Σ rrdiagrxr
σσσσσσ 
and ( ) ( ) riAAAA H
i
H
ii
,,2,10: =>== λλσ
Those equations can be rewritten as: [ ] [ ]
[ ] [ ]0
0
1121
1121
Σ=
Σ=
VVVAA
UUUAA
H
H
or
0
0
2
2
111
111
=
=
Σ=
Σ=
VAA
UAA
VVAA
UUAA
H
H
H
H
H
U2
H
V2
( )
( ) 00
00
22222
22222
=→==
=→==
VAVAVAVAAV
UAUAUAUAAU
HHH
HHHHHH
Singular Values

18
SOLO Matrices
Proof of Lemma 6 (continue – 1):
02 =UAH
The columns of U2 form an orthonormal basis for the nullspace of AH
:
( ) ( ) ( )HH
AANUR ker2 ==
02 =VA The columns of V2 form an orthonormal basis for the nullspace of A:
( ) ( ) ( )AANVR ker2 ==
111 Σ= UUAA H
The columns U1 of form an orthonormal basis for the column space of A:
( ) ( )ARUR =1
111 Σ= VVAAH
The columns of V1 form an orthonormal basis for the column space of AH
:
( ) ( )H
ARVR =1
( )H
AR
( )AN
( )H
AN
( )AR
xAy =
11 yAx H
=
( )Adomxmx ∈1
11mx
x
( )Acodomy nx
∈11
1nx
y
Singular Values

19
SOLO Matrices
( )H
AR
( )AN
( )H
AN
( )AR
xAy =
11 yAx H
=
( )Adomxmx ∈1
11mx
x
( )Acodomy nx
∈11
1nx
y
From
( )
riuAuA
AA
v i
H
i
i
H
H
i
i
,,2,1
11
: ===
σλ
we have [ ] [ ]














=
r
r
H
r uuuAvvv
σ
σ
σ
/100
0/10
00/1
2
1
2121





or 1
111
−
Σ= UAV H
11
1
1
2
11
1
111
2
111
Σ=ΣΣ=Σ= −
Σ=
−
UUUAAVA
UUAA
H
H
riuuuAAvA iiii
i
i
H
i
i
,,2,1
11 2
==== σσ
σσ
from which 1
111
−
Σ= VAU rivAu i
i
i
,,2,1
1
==
σ
111
Σ=VAU
H
Singular Values

20
SOLO Matrices
( )H
AR
( )AN
( )H
AN
( )AR
xAy =
11 yAx H
=
( )Adomxmx ∈1
11mx
x
( )Acodomy nx
∈11
1nx
y
Using and let compute AH
A V111
Σ=VAU
H
02 =VA
( )
( )
[ ] ( )
( ) ( ) ( )
( )
( ) ( ) ( ) 






 Σ
=








=








=
−−−
−
−−−
−
−
−
rmxrnxrrn
rmrx
nxm
H
nxm
H
nxm
H
nxm
H
nxmH
H
mxmnxm
H
nxn
rxr
rmmxxrrnmxrxrrn
rmmxrxnmxrrxn
rmmxmxr
xnrn
rxn
VAUVAU
VAUVAU
VVA
U
U
VAU
00
01
2212
2111
21
2
1
From this equation we obtain:
( )
[ ] ( )
( ) ( ) ( ) ( )
∑=
−−−
−
=Σ=















 Σ
=
−
−
r
i
H
iii
H
H
H
rmxrnxrrn
rmrx
nxm
vuVU
V
V
UUA
rxmrxrnxr
xmrm
rxmrxr
rnnxnxr
1
111
2
11
21
00
0
σ
Singular Values
Table of Contents

21
SOLO Matrices
Let perform the following transformations in
the domain and codomain :
Geometric Interpretation of Singular Values
Suppose, first, that A is square and r = n = m, and consider the spherical hypersurface
in the domain of A for which:
1v 1x
2
x
2
v
1=x
111 vAv =σ
111
xAx =λ
222 xAx =λ
222
vAv =σ
The Indicator Ellipsoid of a 2 x 2 Matrix11
11
nxnxnnx
mxmxmmx
Uy
Vx
η
ς
=
=
Because y = A x: ( ) ςςςη Σ=Σ=== UVVUVAUy H
From which: 11 mxnxmnx
ςη Σ=
1
1
22
2
2
2
===== ∑=
r
i
i
HHH
VVxxx ςςςς
From we have and the mapping of the spherical
hypersurface, in the codomain of A is the hypersurface of an ellipsoid:
11 mxnxmnx
ςη Σ= iii
σης /=
1
1
2
=







∑=
r
i i
i
σ
η
This ellipsoid is called the indicator ellipsoid of A and the singular values are
the lengths of the principal axes of this ellipsoid.
Singular Values

22
SOLO Matrices
If the square matrix A is singular, i.e., r < n = m, the indicator ellipsoid shrinks
to zero in the directions of the principal axes vi for which σi = 0. In this case:
Geometric Interpretation of Singular Values (continue – 1)
01 1
1
2
===<







+
=
∑ nr
r
i i
i
ηη
σ
η

If the general case of nonsquare matrices with r < n ≠ m, if we choose the
cylindrical hypersurface that has a circular hypersurface projection in R (AH
):
01 1
1
2
==== +
=
∑ mr
r
i
i
ςςς 
then its mapping will be the surface of the ellipsoid in R (A).
01 1
1
2
====







+
=
∑ nr
r
i i
i
ηη
σ
η
 ( )H
AR
( )AN
( )H
AN
( )AR
Singular Values
Table of Contents

23
SOLO Matrices
(1) The maximum singular value of Anxm is:
[ ] [ ]
2
2
02121
maxmaxmaxmax:
22 x
xA
xAxAAA
xxx
i
i ≠=≤
==== σσ
(2) The minimum singular value of Anxm is:
[ ] [ ]
2
2
02121
minminminmin:
22 x
xA
xAxAAA
xxx
i
i ≠=≤
==== σσ
Proof of (1) and (2)
Using x = V ζ we can write:
( ) ∑=
=
=Σ=Σ===
m
i
ii
H
xV
HHHHH
xVVxxAAxxAxAxA
1
22222
2
ςσςς
ς
1
1
22
2
≤==== ∑=
= m
i
i
H
xV
HHH
xVVxxxx ςςς
ς
To obtain the maximum/minimum of that satisfy the condition
we construct the Hamiltonian by adjoin the constraint to the extremum problem:
2
2
xA 1
2
2
≤x
( ) 





−+±= ∑∑ ==
1:,
1
2
1
22
m
i
i
m
i
ii
H ςλςσλς
+ for maximum
- for minimum
Singular Values

24
SOLO Matrices
Proof of maximum/minimum singular value of Anxm (continue – 1)
The necessary conditions for extremum are:
( ) 





−+±= ∑∑ ==
1:,
1
2
1
22
m
i
i
m
i
ii
H ςλςσλς
+ for maximum
- for minimum
( ) mi
H
ii
i
,,2,102
2
==+±=
∂
∂
ςλσ
ς
Kuhn-Tucker
Condition
( ) mi
minimumfor
maximumforH
i
i
,,2,1
0
0
2
2
2
2
=



>
<
+±=
∂
∂
λσ
ς
Maximization problem solution: ( ) [ ]AH 22
1,max σσλς ==
with
0
10,1
2
1
1
2
21
<−=
=⇒==== ∑=
σλ
ςςςς
m
i
im

Kuhn-Tucker Condition
Minimization problem solution: ( ) [ ]AH m
22
,min σσλς ==
with
0
10,1
2
1
2
11
≥=
=⇒==== ∑=
−
m
m
i
imm
σλ
ςςςς 
Kuhn-Tucker Condition
Singular Values

25
SOLO Matrices
Proof of maximum/minimum singular value of Anxm (continue – 2)
For any x ≠ 0 we have:
[ ] [ ] ( ) 0max
max
2
2
2
2
2
≤
−
=−=−
xx
xIAAx
AA
xx
xAAx
A
x
xA
H
HH
H
H
HH
λ
λσ
The inequality holds because (AH
A-I λmax [AH
A]) is non-positive definite.
We can see that the equality is satisfied for x = eigenvector (AH
A) that corresponds
to λmax [AH
A], therefore:
[ ] 0max
2
2
0
=








−
≠
A
x
xA
x
σ
In the same way, or any x ≠ 0 we have:
[ ] [ ] ( ) 0min
min
2
2
2
2
2
≥
−
=−=−
xx
xIAAx
AA
xx
xAAx
A
x
xA
H
HH
H
H
HH
λ
λσ
We can see that the equality is satisfied for x = eigenvector (AH
A) that corresponds
to λmin [AH
A], therefore:
[ ] 0min
2
2
0
=








−
≠
A
x
xA
x
σ
[ ]A
x
xA
x
σ=
≠
2
2
0
max
[ ]A
x
xA
x
σ=
≠
2
2
0
min
Singular Values

26
SOLO Matrices
(3) is a norm of Anxm, because it satisfies the norm properties:[ ]Aσ
(3.1) is non-negative and if and only if A = 0.
Proof of (3.1):
From Lemma 3:
[ ]Aσ [ ] 0=Aσ [ ] 00 =⇔= AAσ
(3.2) Multiplication by a complex constant α: [ ] [ ]AA σαασ =
(3.3) Triangle Inequalities:
(3.4) Schwarz Inequality:
[ ] [ ] [ ] [ ] [ ]BABABA σσσσσ +≤+≤−
[ ] [ ] [ ]BABA σσσ ≤
[ ] 0≥Aσ
[ ] 000 =Σ=⇔=Σ⇔= H
VUAAσ
Proof of (3.2):
[ ] ( ) [ ]AAAA
xx
σαααασ ===
≤≤ 2121 22
maxmax
Singular Values

27
SOLO Matrices
(3.1) is non-negative and if and only if A = 0.[ ]Aσ [ ] 0=Aσ [ ] 00 =⇔= AAσ
[ ] [ ] [ ] [ ] [ ]BABABA σσσσσ +≤+≤−
[ ] [ ] [ ]BABA σσσ ≤
Proof of (3.3): [ ] ( ) ( )
[ ] [ ]BAxBxA
xBxAxBABA
xx
xx
σσ
σ
+=+≤
+≤+=+
≤≤
≤≤
2121
22121
22
22
maxmax
maxmax
[ ] ( )[ ] [ ] [ ]BBABBAA σσσσ ++≤−+=From which
[ ] [ ] [ ]BABA +≤− σσσ
In the same way [ ] [ ] [ ]BAAB +≤− σσσ
[ ] [ ] [ ]BABA +≤− σσσ
Proof of (3.4):
[ ]
( ) ( )
[ ] [ ]BA
x
xB
y
yA
x
xB
xB
xBA
x
xBA
BA
xy
xx
σσ
σ
=≤
==
≠≠
≠≠
2
2
0
2
2
0
2
2
2
2
0
2
2
0
maxmax
maxmax
Hermann Amandus
Schwarz
1843 - 1921
Singular Values

28
SOLO Matrices
(4) The absolute value of the eigenvalues of a square matrix Anxn are bounded between
the minimum and the maximum singular values:
[ ] [ ] [ ] niAAA i
,,2,1 =≤≤ σλσ
Proof of (4):
We have: [ ] [ ] 0
2
2
≠∀≤≤ xA
x
xA
A σσ
If xi is any normalized eigenvector: A xi = λi xi, then
ni
x
x
x
x
x
xA
i
i
ii
i
ii
i
i
,,2,1
2
2
2
2
2
2
==== λ
λλ
Therefore: [ ] [ ] [ ] niAAA i
,,2,1 =≤≤ σλσ
Singular Values

29
SOLO Matrices
(5) A square matrix Anxn is singular iff its minimal singular value is zero.
[ ] 0=⇔ ASingularA σ
Proof of (5):
( ) [ ] [ ]AAVdiagUA n
H
n
σσσσσσσσ =≥≥≥==  2121
,,,
Therefore: [ ] [ ] [ ] 00det ==⇔=⇔ AAASingularA nσσ
( )[ ] [ ] ∏=
==
n
i
i
H
n VdiagUA
11
21
1
det,,,detdetdet σσσσ


(6) For a nonsingular square matrix Anxn we have
[ ]
[ ]
[ ]
[ ]11
1
&
1
−−
==⇔
A
A
A
ArNonsingulaA
σ
σ
σ
σ
Proof of (6):
( ) ( ) H
n
H
n UdiagVAVdiagUA σσσσσσ /1,,/1,/1,,, 21
1
21  =⇒= −
[ ] [ ] [ ] [ ]1
1
1
21
/1/10 −−
=≥≥=⇒>=≥≥≥= AAAA nn
σσσσσσσσσ 
Hence: [ ]
[ ]
[ ]
[ ]11
1
&
1
−−
==
A
A
A
A
σ
σ
σ
σ
Singular Values

30
SOLO Matrices
(7) If the square matrix (A+B) is singular then the maximum singular values of A
and of B are greater or equal than the minimum singular value of B and A,
respectively. The opposite is not true.
( ) [ ] [ ] [ ] [ ]ABBASingularBA σσσσ ≥≥⇒+ &
Proof of (7):
If (A+B) is singular, there exists a normalized eigenvector u (║u║2=1), s.t.:
( ) 22
0 uBuAuBuAuBA =⇒−=⇒=+
From this equation we obtain:
[ ] [ ]BxBuBuAxAA
xx
σσ =≥=≥=
≤≤ 212221 22
minmax
[ ] [ ]AxAuAuBxBB
xx
σσ =≥=≥=
≤≤ 212221 22
minmax
To prove that the opposite is not true, consider a counterexample:
[ ] [ ] [ ] [ ] 15&34
30
05
10
04
=>==>=




−
=





= ABBABA σσσσ
The right side is satisfied, but is nonsingular.( ) 




−
=+
40
01
BA
Singular Values

31
SOLO Matrices
Proof of (8):
(8) A sufficient condition that the square matrix (A+B) is nonsingular is:
We just proved:
( ) [ ] [ ] [ ] [ ]ABorBAingularonsNBA σσσσ <<⇒+
The proof follows directly from property (7). If (A+B) is singular then
[ ] [ ] [ ] [ ]ABBA σσσσ ≥≥ & ; hence if
then (A+B) is nonsingular.
[ ] [ ] [ ] [ ]ABorBA σσσσ <<
Singular Values

32
SOLO
To prove this we will consider the following three cases:
- (A+B) singular,
- (A+B) nonsingular but A and B are singular,
- (A+B) nonsingular but A or B, or both are nonsingular.
Matrices
(9) The minimum singular value of a square matrix (A+B) satisfies the inequalities:
[ ] [ ] [ ] [ ]( ) [ ] [ ] [ ] [ ] [ ]( )ABBABAABBA σσσσσσσσσ ++≤+≤−− ,min,max
Proof of (9):
(9.1) - (A+B) singular
According to property (5) [ ] 0=+ BAσ
Since (A+B) is singular use property (7)
[ ] [ ] [ ] [ ] [ ]BABAAB +=≤−⇒≥ σσσσσ 0
[ ] [ ] [ ] [ ] [ ]BAABBA +=≤−⇒≥ σσσσσ 0
This completes the proof when (A+B) is singular.
Singular Values

33
SOLO
(9.2) - (A+B) nonsingular but A and B are singular,
Matrices
Proof of (9) (continue – 1):
( )  ( ) 22
0
uBuBAuBuBuAuBA =+⇒=+=+
If A is singular, ,there exists a normalized eigenvector u (║u║2=1), s.t. A u=0:[ ] 0=Aσ
[ ] ( ) ( ) [ ] [ ] [ ]ABBxBuBuBAxBABA
xx
σσσσ +==≤=+≤+=+
≤≤ 212221 22
maxmin
and
[ ] [ ] [ ] [ ] [ ]BABABA σσσσσ +≤+<−
In the same way for (A+B) nonsingular and B singular:
[ ] [ ] [ ] [ ] [ ]ABBAAB σσσσσ +≤+<−
This completes the proof when (A+B) is nonsingular but A and B are singular.
Singular Values

34
SOLO
(9.3) - (A+B) nonsingular but A or B, or both are nonsingular.
Matrices
BAC +=:Suppose that (A+B) and A are nonsingular, and define:
Pre-multiply by C-1
and post-multiply by A-1
:
1111 −−−−
+= ABCCA
Let take any norm of this equation and write triangle an Schwarz inequalities:
BACABC
ABCCAABCC
1111
1111111
−−−−
−−−−−−−
≤
−≤≤−
BACCABACC 1111111 −−−−−−−
−≤≤−
( )
B
ABA
B
A
+≤
+
≤− −−− 111
111
Using property (3), we can define , and because property (6) the previous
equation is equivalent to:
[ ]** σ=
[ ] [ ] [ ] [ ] [ ]BABABA σσσσσ +≤+<−
If B is nonsingular in the same way we can prove that:
[ ] [ ] [ ] [ ] [ ]ABBAAB σσσσσ +≤+<−
This completes the proof when (A+B) is nonsingular but A or B, or both are nonsingular.
Singular Values

35
SOLO
Using this and property (3.3):
Matrices
(10) If the square matrix A is a big matrix relative to the square matrix B, then (A+B)
can be approximated by A:
[ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ]ABAABABABAIf σσσσσσσσ ≈+≈+≤+⇒>> &
Proof of (10):
We have: [ ] [ ] [ ] [ ]BBAA σσσσ ≥>>≥
[ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ]ABAABABABAA σσσσσσσσσ ≈+⇒≈+≤+≤−≈
[ ] [ ] [ ] [ ] [ ]BABABA σσσσσ +≤+≤−
Using: and property (9):[ ] [ ] [ ] [ ]BBAA σσσσ ≥>>≥
we have:
[ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ]ABAABABABAA σσσσσσσσσ ≈+⇒≈+≤+≤−≈
Singular Values

36
SOLO Matrices
(11) Multiplicative Inequalities for square matrices:
[ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ]BABABABABABA σσσσσσσσσσ ≤≤≤≤ &
Proof of (11):
The proof is given in the following steps:
(11.1) is the Schwarz inequality of property (3)[ ] [ ] [ ]BABA σσσ ≤
(11.2) prove that :[ ] [ ] [ ]BABA σσσ ≤
If A or B is singular ( or is zero) then A B is singular ( det [A B] =
det [A].
det [B]=0 and ) and we have equality.
[ ]Aσ [ ]Bσ
[ ] 0=BAσ
If A or B is nonsingular then A B is nonsingular ( det [A B] =
det [A].
det [B]≠0 ) and:
( ) ( ) 11111111 −−−−−−−−
≤=== ABABBAABBA
We define , and use the property (6):[ ]** σ= [ ]
[ ]
[ ]
[ ]11
1
&
1
−−
==
A
A
A
A
σ
σ
σ
σ
to obtain: [ ] [ ]
( )
[ ]BA
BABA
BA σσσ ≤≤= −−− 111
111
This result is opposite to Schwarz inequality, proving that is not a norm.[ ]Aσ
Singular Values

37
SOLO Matrices
(11.3) prove that :
If A is singular then:
[ ] [ ] [ ] [ ] [ ]BABAorBA σσσσσ ≤
[ ] [ ] [ ]BABA σσσ ≤= 0
If A is nonsingular then:
[ ] [ ] [ ] [ ]
[ ]
[ ]
[ ]
[ ] [ ]BAB
A
BA
A
BAABAAB σσ
σ
σ
σ
σσσσ ≤⇒=≤= −− 1111
If B is singular then: [ ] [ ] [ ]BABA σσσ ≤= 0
If B is nonsingular then:
[ ] [ ] [ ] [ ] [ ]
[ ]
[ ] [ ] [ ]BABA
B
BA
BBABBAA σσσ
σ
σ
σσσσ ≤⇒=≤= −− 11
Singular Values

38
SOLO Matrices
(11.4) prove that :
If A or B are singular then A B is singular, and:
[ ] [ ] [ ] [ ] [ ]BABAorBA σσσσσ ≤
[ ] [ ] [ ] [ ] [ ]( )BAorBABA σσσσσ ≤= 0
If B is nonsingular then:
[ ] [ ] [ ] [ ] [ ]
[ ]
[ ] [ ] [ ]BABA
B
BA
BBABBAA σσσ
σ
σ
σσσσ ≤⇒=≥= −− 11
We also have: [ ] [ ] [ ] [ ] [ ] [ ] [ ]
[ ]
[ ] [ ] [ ]BABA
B
BA
BBABBABBAA
σσσ
σ
σ
σσσσσσ
≤⇒
=≥≥= −−− 111
If A is nonsingular then:
[ ] [ ] [ ] [ ] [ ]
[ ]
[ ] [ ] [ ]BABA
A
BA
BAABAAB σσσ
σ
σ
σσσσ ≤⇒=≥= −− 11
We also have: [ ] [ ] [ ] [ ] [ ] [ ] [ ]
[ ]
[ ] [ ] [ ]BABA
A
BA
BAABAABAAB
σσσ
σ
σ
σσσσσσ
≤⇒
=≥≥= −−− 111
q.e.d.
Singular Values

39
SOLO Matrices
(12) Any unitary matrix U (U UH
= UH
U = I) has all the singular values equal to 1.
Proof of (12):
[ ] [ ] [ ] iIUUU i
H
ii ∀=== 12/12/1
λλσ
(13) If U is a unitary matrix (U UH
= UH
U = I) then:
Proof of (13):
[ ] [ ] [ ] iAUAAU iii ∀== σσσ
[ ] ( ) ( )[ ] [ ] [ ] [ ] iAAAAUUAAUAUAU i
H
i
HH
i
H
ii ∀==== σλλλσ 2/12/12/1
[ ] ( ) ( )[ ] [ ] [ ] [ ] [ ] iAAAAAUUUAAUUAUAUA i
H
i
HH
i
HH
i
H
ii ∀===== σλλλλσ 2/12/12/12/1
q.e.d.
Singular Values
q.e.d.

40
SOLO Matrices
Properties of Singular Values - Summary
(4) The absolute value of the eigenvalues of a square matrix Anxn are bounded between
the minimum and the maximum singular values:
[ ] [ ] [ ] niAAA i
,,2,1 =≤≤ σλσ
(3.1) is non-negative and if and only if A = 0.[ ]Aσ [ ] 0=Aσ [ ] 00 =⇔= AAσ
[ ] [ ] [ ] [ ] [ ]BABABA σσσσσ +≤+≤−
[ ] [ ] [ ]BABA σσσ ≤
(1) The maximum singular value of Anxm is:
[ ] [ ]
2
2
02121
maxmaxmaxmax:
22 x
xA
xAxAAA
xxx
i
i ≠=≤
==== σσ
(2) The minimum singular value of Anxm is:
[ ] [ ]
2
2
02121
minminminmin:
22 x
xA
xAxAAA
xxx
i
i ≠=≤
==== σσ
Singular Values

41
SOLO Matrices
Properties of Singular Values – Summary (continue – 1)
(5) A square matrix Anxn is singular iff its minimal singular value is zero.
[ ] 0=⇔ ASingularA σ
(6) For a nonsingular square matrix Anxn we have
[ ]
[ ]
[ ]
[ ]11
1
&
1
−−
==⇔
A
A
A
ArNonsingulaA
σ
σ
σ
σ
(8) A sufficient condition that the square matrix (A+B) is nonsingular is:
( ) [ ] [ ] [ ] [ ]ABorBAingularonsNBA σσσσ <<⇒+
Singular Values

42
SOLO Matrices
Properties of Singular Values – Summary (continue – 2)
(12) Any unitary matrix U (U UH
= UH
U = I) has all the singular values equal to 1.
(13) If U is a unitary matrix (U UH
= UH
U = I) then: [ ] [ ] [ ] iAUAAU iii ∀== σσσ
(10) If the square matrix A is a big matrix relative to the square matrix B, then (A+B)
can be approximated by A:
[ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ]ABAABABABAIf σσσσσσσσ ≈+≈+≤+⇒>> &
Singular Values
Table of Contents

43
SOLO Matrices
Householder Transformation
nˆ
( )xnn T 
ˆˆ
( )xnn T 
ˆˆ
x

'x

O A
We want to compute the reflection of
over a plane defined by the normal ( )1ˆˆˆ =nnn T
x

From the Figure we can see that:
( ) ( ) xHxnnIxnnxx TT 
=−=−= ˆˆ2ˆˆ2'
1ˆˆˆˆ2: =−= nnnnIH TT
We can see that H is symmetric:
( ) HnnInnIH TTTT
=−=−= ˆˆ2ˆˆ2
In fact H is also a rotation of around OA so it must be orthogonal, i.e.
HT
H=H HT
=I.
x

( ) ( )  InnnnnnInnInnIHHHH TTTTTT
=+−=−−== ˆˆˆˆ4ˆˆ4ˆˆ2ˆˆ2
1
†
of Amxn .
Table of Contents
Alston Scott Householder
1904 - 1993

44
SOLO Matrices
The same result is obtained if we compute α that minimizes:
a

b

P
a

α
ab

α−p

We want to find such that ( )pba

−⊥ap

α=
Projection of a vector on a vector .b

a

or: ( ) ( ) ( ) baaaabapba TTTT
 1
0
−
=⇒−=−= αα
and: ( ) ( )[ ] bPbaaaabaaaaap TTTT

====
−− 11
α
( )[ ]TT
aaaaP
 1
:
−
= Projection Matrix
( ) ( ) ( )aababbabababd TTT
T  2
2
2
2minminminmin ααααα αααα
+−=−−=−=
( )
( )
( ) baaa
aa
d
aaba
d
TT
T
TT



1
min
2
22
2
0
022
−
=⇒







>=
∂
∂
=+−=
∂
∂
α
α
α
α
Properties of Projection Matrix
(1) P is idempotent P2
= P
(2) P is symmetric PT
= P
( ) cbcPIcPcbP

,∀−=−⊥
Proof:
( ) ( ) ( ) cbcPIPbcPIbP TT
T 
,0 ∀−=−=
( ) 0=− PIPT
Hence: PPP TT
= ( ) PPPPP TTT
== 2
PPPPP TT
===
b

bP

cP

cP
c

 −
c

†
of Amxn .
Table of Contents

45
SOLO
Note:
If A and b were real, instead of H (transpose & complex conjugate) we
have only T (transpose).
Matrices
Given: Amxn of rank (Amxn) = r ≤ min (m,n) and 1mx
b

Find: such that is minimal1nx
x

11 mxnxmxn
bxAd

−=
If the solution is not unique choose such that is minimal1nx
x

1nx
x

Solution:
The minimum is obtained when
( ) ( )
( )







>=
∂
∂
=−=−=−=
∂
∂
0
0
2
22
2
11
2
AA
x
d
bAxAAbxAAbxA
x
d
H
HHH
mxnxmxn



( ) bAAAx HH
 1−
=
A unique solution exists if AH
A is positive definite, or rank (AH
A) = n, or det|AH
A| ≠ 0
( ) ( )1111
2
11
2
mxnxmxn
H
mxnxmxnmxnxmxn
bxAbxAbxAd

−−=−=Analytic:
†
of Amxn .
111
min&min
1
nxmxnxmxn
x
xbxAd
nx

 −=

46
SOLO Matrices
Given: Amxn of rank (Amxn) = r ≤ min (m,n) and 1mx
b

Find: such that is minimal1nx
x

11 mxnxmxn
bxAd

−=
If the solution is not unique choose such that is minimal1nx
x

1nx
x

Solution (continue – 1):
Geometric:
We have A x∈ R (A) for all x ∈ domain (A).
We want to find x0 ∈ domain (A), such that
is normal to A x.0
xAbpb

−=−
( ) ( ) ( )AdomainxxAbpbxA ∈∀−=−⊥

0
( ) ( ) ( ) ( )AdomainxxAAbAxxAbxA HHHH
∈∀−=−=

00
0
Hence: 00
=− bAxAA HH

( )H
AR
( )AN
( )H
AN
( )AR
xA

( )Adomx ∈0

( )AcodomY ∈
Null Space of A
Ker (A)
span by VA2
H
Row Space of A
span by VA1
H
Column Space of A
span by UA1
Left Null Space of A
span by UA2
b

R
x0

Nx0

0
xAp

=
pb

−
xA

( )Adomx ∈

0
xA

†
of Amxn .
111
min&min
1
nxmxnxmxn
x
xbxAd
nx

 −=

47
SOLO Matrices
Let decompose as0
x

( )H
AR
( )AN
( )H
AN
( )AR
xA

( )Adomx ∈0

( )AcodomY ∈
Null Space of A
Ker (A)
span by VA2
H
Row Space of A
span by VA1
H
Column Space of A
span by UA1
span by UA2
b

R
x0

N
x0

0xAp

=
pb

−
xA

( )Adomx ∈

0xA

( )
( ) NR
N
H
R
NR
xx
AofspaceNullANx
AofspaceRowARx
xxx 00
0
0
000




⊥



∈
∈
+=
Therefore:
RNR
xAxAxAxAp 0
0
000



=+==
Hence if:
(a) N (A) = 0 or
(b) The rows of A are linearly dependent or
(c) rank (A) = r < m
(d) AH
A is singular
there are a infinity of solutions NRNR
xxxxx 00000

⊥+=
The norm of is:0x

NR
xx
NR
xxxxx
NR
00000
00 

+=+=
⊥
Hence: 0&min 000
== NR
xxx

†
of Amxn .
111
min&min
1
nxmxnxmxn
x
xbxAd
nx

 −=

48
SOLO Matrices
†
of Amxn .
( )T
AR
( )AN
( )H
AN
( )AR
xA

( )Adomx ∈0

( )AcodomY ∈
Null Space of A
Ker (A)
span by VA2
T
Row Space of A
span by VA1
T
Column Space of A
span by UA1
span by UA2
b

R
x0

N
x0

0xAp

=
pb

−
xA

( )Adomx ∈

0xA

bAx R
 +
=0
0&min 000
== NR
xxx

Define the Linear Transformation (Matrix),
that gives from , as the Pseudoinverse
of A. (A is the direct transformation that gives
from :
Rx0

b

p

x

xAp

=
bAx †
R

=0
A†
is called Moore-Penrose Pseudoinverse Matrix,
because was defined independently by E.H.Moore in 1920
and Roger Penrose in 1955.
Eliakim Hastings
Moore
1862 - 1932
Roger Penrose
1931 -
111
min&min
1
nxmxnxmxn
x
xbxAd
nx

 −=
Table of Contents

49
SOLO Matrices
†
of Amxn .a

( )H
AR
( )AN
( )H
AN
( )AR
xA

( )Adomx ∈0

( )AcodomY ∈
Null Space of A
Ker (A)
span by VA2
H
Row Space of A
span by VA1
H
Column Space of A
span by UA1
span by UA2
b

R
x0

N
x0

0xAp

=
pb

−
xA

( )Adomx ∈

0xA

bAx †
R

=0
bAx †
R

=0
Perform Singular Value Decomposition (S.V.D.) of Amxn:
where ( ) 0,,, 21211 >≥≥≥=Σ rrA diagrxr
σσσσσσ 
UAmxm and VAnxn are unitary matrices, i.e.:
( )
[ ] ( )
( ) ( ) ( ) ( ) 














Σ
=Σ=
−
−
−−−
−
H
A
H
A
rnxrmxrrm
rnrxA
AA
H
AAAmxn
xnrn
rxnrxr
rmmxmxrnxnmxnmxm
V
V
UUVUA
2
11
21
00
0

[ ]
( ) ( )
[ ] H
AA
H
AA
H
AAH
A
H
A
AA
rmxrm
rxr
AAH
A
H
A
A
H
A UUUUUU
U
U
UU
I
I
UU
U
U
UU =+=





=





=





=
−−
2211
2
1
2121
2
1
0
0
[ ]
( ) ( )
[ ] H
AA
H
AA
H
AAH
A
H
A
AA
rnxrn
rxr
AAH
A
H
A
A
H
A VVVVVV
V
V
VV
I
I
VV
V
V
VV =+=





=





=








=
−−
2211
2
1
2121
2
1
0
0

50
SOLO Matrices
( )H
AR
( )AN
( )H
AN
( )AR
xA

( )Adomx ∈0

( )AcodomY ∈
Null Space of A
Ker (A)
span by VA2
H
Row Space of A
span by VA1
H
Column Space of A
span by UA1
span by UA2
b

R
x0

N
x0

0xAp

=
pb

−
xA

( )Adomx ∈

0xA

bAx †
R

=0
Since the norm is invariant to the product of orthogonal matrices
( ) bUxVbxVUUbxVUbxA HHHHH

−Σ=−Σ=−Σ=−
Introduce the new unknown: R
H
N
H
R
HH
xVxVxVxVy

=+==:
But 
( ) 0
0
=⇒Σ=+Σ=+= N
H
R
H
N
H
R
H
NR xVxVUxVxVUxAxAxA

RR
I
HH
RR
HH
xxVVxxVxVy



=








===
2/1
( )
( ) ( ) ( )
bUybUybxA H
rnxrmxrrm
r
rnrx
y
H
yx







 −
















=−Σ=−
−−−
−
00
0
0
0
minminmin
1
σ
σ
†
of Amxn .

51
SOLO Matrices
( )H
AR
( )AN
( )H
AN
( )AR
xA

( )Adomx ∈0

( )AcodomY ∈
Null Space of A
Ker (A)
span by VA2
H
Row Space of A
span by VA1
H
Column Space of A
span by UA1
span by UA2
b

Rx0

Nx0

0xAp

=
pb

−
xA

( )Adomx ∈

0xA

bAx †
R

=0
Therefore:
R
HH
xVxVy

==:
RR
I
HT
RR
HH
xxVVxxVxVy



=








===
2/1
( )
( ) ( ) ( )
bUybUybxA H
rnxrmxrrm
r
rnrx
y
H
yx







 −
















=−Σ=−
−−−
−
00
0
0
0
minminmin
1
σ
σ
( )
( ) ( ) ( )
( )




















+
















= −
−−−
−








any
xrm
rx
H
rmxrnxrrn
r
rmrx
XbUy 1
1
1
0
00
/10
0
0/1
σ
σ
R
xyx 0
minmin

==
( )
( ) ( ) ( )
R
HH
rmxrnxrrn
r
rmrx
xVbUy 0
1
00
/10
0
0/1

  






=
















=
+
Σ
−−−
−
σ
σ
bAbUVx †H†
R

=Σ=0
H††
UVA Σ=
†
of Amxn .

52
SOLO Matrices
( )H
AR
( )AN
( )H
AN
( )AR
xA

( )Adomx ∈0

( )AcodomY ∈
Null Space of A
Ker (A)
span by VA2
H
Row Space of A
span by VA1
H
Column Space of A
span by UA1
span by UA2
b

R
x0

N
x0

0
xAp

=
pb

−
xA

( )Adomx ∈

0xA

bAx †
R

=0
Where:
( )
( ) ( ) ( ) 

















=Σ
−−−
−
−
=
rmxrnxrrn
r
rmrx
†
nxm
00
0
0
0
:
1
1
1





σ
σ
H
mxm
†
nxmnxn
†
nxm UVA Σ=
†
of Amxn .
( )
[ ] ( )
( ) ( ) ( ) ( )
H
AAAH
A
H
A
rmxrnxrrn
rmrxA
AnA
†
nxm rxmrxrnxr
xmrm
rxmrxr
rnnxmxr
UV
U
U
VVA 1
1
11
2
1
1
1
21
00
0
: −
−−−
−
−
Σ=















Σ
=
−
−











=Σ
−
=
1
1
1
1
0
0
:
r
1-
A rxr
σ
σ



Table of Contents

53
SOLO Matrices
†
of Amxn .
( )
( ) ( ) ( )
( )
( ) ( ) ( )
( )
( ) ( ) ( ) 







=


































=ΣΣ
−−−
−
−−−
−
−−−
−
−
=
rnxrnxrrn
rnrxrxr
rnxrmxrrm
r
rnrx
rmxrnxrrn
r
rmrx
mxn
†
nxm
I
00
0
00
0
0
0
00
0
0
0 1
1
1
1










σ
σ
σ
σ
( )
( ) ( ) ( )
( )
( ) ( ) ( )
( )
( ) ( ) ( ) 







=


































=ΣΣ
−−−
−
−−−
−
−
=
−−−
−
rmxrmxrrm
rmrxrxr
rmxrnxrrn
r
rmrx
rnxrmxrrm
r
rnrx
†
nxmmxn
I
00
0
00
0
0
0
00
0
0
0
1
1
11










σ
σ
σ
σ
( ) ( )
( ) ( ) ( )
†
nxmmxn
rmxrmxrrm
rmrxrxr††
nxmmxn
I
ΣΣ=








=ΣΣ
−−−
−
00
0
( ) ( )
( ) ( ) ( )
mxn
†
nxm
rnxrnxrrn
rnrxrxr†
mxn
†
nxm
I
ΣΣ=








=ΣΣ
−−−
−
00
0
Using the definition of the Pseudoinverse we can see that

54
SOLO Matrices
†
of Amxn .
( )
( ) ( ) ( )
( )
( ) ( ) ( )
( )
( ) ( ) ( )
†
nxm
rmxrnxrrn
r
rmrx
rmxrnxrrn
r
rmrx
rnxrnxrrn
rnrxrxr†
nxmmxn
†
nxm
I
Σ=


















=


























=ΣΣΣ
−−−
−
−
=
−−−
−
−
=
−−−
−
00
0
0
0
00
0
0
0
00
0 1
1
1
1
1
1










σ
σ
σ
σ
( )
( ) ( ) ( )
( )
( ) ( ) ( )
( )
( ) ( ) ( )
mxn
rnxrmxrrm
r
rnrx
rnxrmxrrm
r
rnrx
rmxrmxrrm
rmrxrxr
mxn
†
nxmmxn
I
Σ=
















=
























=ΣΣΣ
−−−
−
−−−
−
−−−
−
00
0
0
0
00
0
0
0
00
0
11










σ
σ
σ
σ
( ) ( ) ( ) ( ) †
nxmmxn
H†H††
Def†
†H††H†H††
nxmmxn
AAUUUUUUUVVUAA =ΣΣ=ΣΣ=ΣΣ=ΣΣ=
( ) ( ) ( ) ( ) mxn
†
nxm
H†H††
Def†
†H††HH††
mxn
†
nxm
AAVVVVVVVUUVAA =ΣΣ=ΣΣ=ΣΣ=ΣΣ=
Also:
Let check the same operations for Matrix A †
( ) ( ) ( ) ( ) mxn
HH†HH†H
mxn
†
nxmmxn
AVUVUVUUVVUAAA =Σ=ΣΣΣ=ΣΣΣ=
( ) ( ) ( ) ( ) †
nxm
H†H††H†HH††
nxmmxn
†
nxm
AUVUVUVVUUVAAA =Σ=ΣΣΣ=ΣΣΣ=

55
SOLO Matrices
†
of Amxn .
- Summary)
( ) †††
nxmmxnnxmmxn AAAA =
( ) mxn
†
nxm
†
mxn
†
nxm AAAA =
mxnmxn
†
nxmmxn AAAA =
†
nxm
†
nxmmxn
†
nxm
AAAA =
1
2
3
4
Table of Contents

56
SOLO Matrices
†
of Amxn .
bPbAAxAp †
R

1
===1
( )H
AR
( )AN
( )H
AN
( )AR
Rx

( )Adomx∈
( )Acodomb ∈

Null Space of A
Ker (A)
span by VA2
H
Row Space of A
span by VA1
H
Column Space of A
span by UA1
span by UA2
b

bAx †
R

=
( )
R
†
xA
bAAp


=
=
R
xAp

=
P1 is a projection matrix because
( ) ( ) ( )



=ΣΣ=ΣΣ==
===
11
1
2
1
PUUUVVUAAP
PAAAAAAP
HH†HH†HH†H
†††
P1=A A†
projects into column space of A, R (A)b

H†H†H†
UUUVVUAAP ΣΣ=ΣΣ==:1

57
SOLO Matrices
†
of Amxn .
2
( )H
AR
( )AN
( )H
AN
( )AR
R
x

( )Adomx ∈

( )Acodomb ∈

Null Space of A
Ker (A)
span by VA2
H
Row Space of A
span by VA1
H
Column Space of A
span by UA1
span by UA2
b

bAx †
R

=
( )
R
†
xA
bAAp


=
=
RxAp

=
( )bAAIpb †

−=−
( ) 0=− pbA† 
pAx †
R

=
P2=(I - A A †
) is a projection matrix because
( ) ( )
( ) ( )




=−=−=−=−=
=+−−=−−=
21112
2
2
2
PPIPIPIAAIP
PAAAAAAAAIAAIAAIP
HHH†H
†
A
†††††

Because , is the
projection of into .
( ) ( )H
ANAR ⊥ pb

−
( )H
ANb

We can see, also, that:
( ) H†H†
†
UIUUUI
AAIPIP
ΣΣ−=ΣΣ−=
−=−= 12 :
( ) ( ) 00



==








−=−=− bbAAAAbAAIApbA
†
A
†††††† pAbAx ††
R

==
( ) ( ) bPbAAIpb †

2
=−=−

58
SOLO Matrices
†
of Amxn .
3 ( )
( ) NR
N
H
R
NR
xx
AofspaceNullANx
AofspaceRowARx
xxx




⊥



∈
∈
+=
( ) ( ) xPxAAxAApAbAx ††††
R

3
=====
P3=A †
A is a projection matrix of in R (AH
)x

( ) ( )
( ) ( ) ( )




=ΣΣ=ΣΣ==
====
33
3
2
3
PVVVUUVAAP
PAAAAAAAAAAP
HH†HHH†H†H
†
A
††††
†

4 ( ) ( ) xPxAAIxAAxxxx ††
RN

4
=−=−=−=
P4=I-A †
A is a projection matrix of in N (A)x

( ) 
( ) ( )




=−=−=−=
=+−=−=
43333
4
2
33
2
3
2
4
3
2
PPIPIPIP
PPPIPIP
HHHH
P
( )H
AR
( )AN
( )H
AN
( )AR
( )Adomx ∈

( )Acodomb ∈

Null Space of A
Ker (A)
span by VA2
H
Row Space of A
span by VA1
H
Column Space of A
span by UA1
span by UA2
b

bAx †
R

=
( )
R
†
xA
bAAp


=
=
R
xAp

=
( )bAAIpb †

−=−
( ) 0=− pbA† 
pAx †
R

=
( ) xAAx †
R

=
( )H
AR
( )AN
( )H
AN
( )AR
( )Adomx ∈

( )Acodomb ∈

Null Space of A
Ker (A)
span by VA2
H
Row Space of A
span by VA1
H
Column Space of A
span by UA1
span by UA2
b

bAx †
R

=
( )
R
†
xA
bAAp


=
=
R
xAp

=
( )bAAIpb †

−=−
( ) 0=− pbA† 
pAx †
R

=
( ) xAAx †
R

=
( ) xAAIx †
N

−=
0=NxA


59
SOLO Matrices
†
of Amxn .
Description of Projections Related to Moore-Penrose Pseudoinverse (Summary)
3 ( ) ( ) ( )H††††
R
ARxPxAAxAApAbAx ∈=====

3
4
( )H
AR
( )AN
( )H
AN
( )AR
( )Adomx ∈

( )Acodomb ∈

Null Space of A
Ker (A)
span by VA2
H
Row Space of A
span by VA1
H
Column Space of A
span by UA1
span by UA2
b

bAx †
R

=
( )
R
†
xA
bAAp


=
=
R
xAp

=
( )bAAIpb †

−=−
( ) 0=− pbA† 
pAx †
R

=
( ) xAAx †
R

=
( ) xAAIx †
N

−=
0=NxA

( ) ( ) ( )ANxPxAAIxAAxxxx ††
RN
∈=−=−=−=

4
( ) ( ) ( )H†
ANbPbAAIpb ∈=−=−

2
2
( )ARbPbAAxAp †
R
∈===

1
1
H††
UUAAP ΣΣ==:1
†
AAIPIP −=−= 12
:
H††
VVAAP ΣΣ==:3
AAIPIP †
−=−= 34 :
Table of Contents

60
SOLO Matrices
†
of Amxn .
Particular case (1) r = n ≤ m:
( )
[ ]
( )
H
A
xnnm
A
AA
H
AAAmxn nxn
nxn
nmmxmxnnxnmxnmxm
VUUVUA







Σ
=Σ=
−
−
0
1
21

(a) rank (Amxn) = n or
(b) columns of Amxn are linear independent or
(c) N (Amxn) = 0 or
(d) Anxm
H
Amxn is nonsingular
This is equivalent to:
where ( ) 0,,, 21211 >≥≥≥=Σ nnA diagnxn
σσσσσσ 
[ ]
( ) ( )
[ ] H
AA
H
AA
H
AAH
A
H
A
AA
nmxnm
nxn
AAH
A
H
A
A
H
A UUUUUU
U
U
UU
I
I
UU
U
U
UU =+=





=





=





=
−−
2211
2
1
2121
2
1
0
0
( )[ ]
( )
H
AAAH
A
H
A
nmnxAA
†
nxm nxmnxnnxn
xmnm
nxm
nxnnxn
UV
U
U
VA 1
1
1
2
11
1 0: −
−
−
Σ=








Σ=
−
( )H
AR
( ) 0=AN
( )H
AN
( )AR
x

b

Row Space of A
span by VA1
H
Column Space of A
span by UA1
span by UA2
( )bAAxAp †

==
pAx † 
=
( )bAAIpb
 +
−=−
xAp

=
bAx †

=
( ) 0=− pbA† 

61
SOLO Matrices
†
of Amxn .
Particular case (1) r = n ≤ n: (continue – 1)
( )[ ]
( )
H
AAAH
A
H
A
nmnxAA
†
nxm nxmnxnnxn
xmnm
nxm
nxnnxn
UV
U
U
VA 1
1
1
2
11
1 0: −
−
−
Σ=








Σ=
−
( )
[ ]
( )
H
A
xnnm
A
AA
H
AAAmxn nxn
nxn
nmmxmxnnxnmxnmxm
VUUVUA







Σ
=Σ=
−
−
0
1
21

[ ] [ ] H
AAA
H
A
A
AAH
A
H
A
AA
H
VVVUU
U
U
VAA 2
1
1
21
2
1
1
0
0 Σ=




Σ






Σ=  ( ) H
AAA
H
VVAA 2
1
1 −−
Σ=
( ) †H
AAA
H
AAA
H
AAA
HH
AUVUVVVAAA =Σ=ΣΣ= −−−
1
1
111
2
1
1
or ( ) H
nxmmxn
H
nxm
†
nxm
AAAA
1−
=
( )H
AR
( ) 0=AN
( )H
AN
( )AR
x

b

Row Space of A
span by VA1
H
Column Space of A
span by UA1
span by UA2
( )bAAxAp †

==
pAx † 
=
( )bAAIpb
 +
−=−
xAp

=
bAx †

=
( ) 0=− pbA† 
We have only one solution that minimize
11 mxnxmxn
bxAd

−=
x

and is given by:
( ) 1
1
11 mx
H
nxmmxn
H
nxmmx
†
nxmnx
bAAAbAx
 −
== Table of Contents

62
SOLO Matrices
†
of Amxn .
(a) rank (Amxn) = m or
(b) rows of Amxn are linear independent or
(c) N (Anxm
H
) = 0 or
(d) AmxnAnxm
H
is nonsingular
This is equivalent to:
where ( ) 0,,, 21211 >≥≥≥=Σ mmA diagmxm
σσσσσσ 
( )
[ ]
( )
H
AAA
H
A
xmmn
A
AnA
†
nxm mxmmxmnxmmxm
mxm
mnnxmxr
UVUVVA 1
11
1
1
21
0
: −
−
−
Σ=







Σ
= −

( )[ ]
( )
H
AAAH
A
H
A
mnmxAA
H
AAAmxn mxnmxmmxm
xnmn
mxn
mxmmxmnxnmxnmxm
VU
V
V
UVUA 11
2
1
1
0 Σ=








Σ=Σ=
−
−
[ ]
( ) ( )
[ ] H
AA
H
AA
H
AAH
A
H
A
AA
mnxmn
mxm
AAH
A
H
A
A
H
A VVVVVV
V
V
VV
I
I
VV
V
V
VV =+=





=





=








=
−−
2211
2
1
2121
2
1
0
0
( )H
AR
( )AN
( ) ( )ARBAN H
∈≡ &0
( )AR
xAb

=
( ) N
†
nxn xxAAI

=−
( ) R
†
xxAA

=
Null Space of A
Ker (A)
span by VA2
H
Row Space of A
span by VA1
H Column Space of A
span by UA1
bAx †
R

= R
xAb

=
b

0=N
xA


63
SOLO Matrices
†
of Amxn .
Particular case (2) r = m ≤ n: (continue – 1)
( )
[ ]
( )
H
AAA
H
A
xmmn
A
AnA
†
nxm mxmmxmnxmmxm
mxm
mnnxmxr
UVUVVA 1
11
1
1
21
0
: −
−
−
Σ=







Σ
= −

( )[ ]
( )
H
AAAH
A
H
A
mnmxAA
H
AAAmxn mxnmxmmxm
xnmn
mxn
mxmmxmnxnmxnmxm
VU
V
V
UVUA 11
2
1
1
0 Σ=








Σ=Σ=
−
−
H
AAA
H
AA
I
A
H
AAA
H
UUUVVUAA
m
2
11111 Σ=ΣΣ=

( ) H
AAA
H
UUAA 2
1
1 −−
Σ=
( ) †H
AAA
H
AAA
H
AAA
HH
AUVUUUVAAA =Σ=ΣΣ= −−− 1
11
2
111
1
or ( ) 1−
= HH†
AAAA
( )H
AR
( )AN
( ) ( )ARBAN H
∈≡ &0
( )AR
xAb

=
( ) N
†
nxn
xxAAI

=−
( ) R
†
xxAA

=
Null Space of A
Ker (A)
span by VA2
H
Row Space of A
span by VA1
H Column Space of A
span by UA1
bAx †
R

= R
xAb

=
b

0=N
xA

We have an infinite number of solutions that minimize
11 mxnxmxn
bxAd

−=
( ) bAAAbAx HH†
R
 1−
==
The solution that minimizes
the norm is given by:
Rx

Rx

Table of Contents

64
SOLO Matrices
X - nxp unknowns with mxp equations
mxpnxpmxn BXA =
where ( ) 0,,, 21211
>≥≥≥=Σ rrA
diagrxr
σσσσσσ 
( )H
AR
( )AN
( )H
AN
( )ARBXA =
11 yAx H
=
( )AdomX ∈
( )AcodomY ∈
1nx
y
Null Space of A
Ker (A)
span by VA2
H
Row Space of A
span by VA1
H
Column Space of A
span by UA1
span by UA2
B
( )
[ ] ( )
( ) ( ) ( ) ( ) 














Σ
=Σ=
−
−
−−−
−
H
A
H
A
rnxrmxrrm
rnrxA
AA
H
AAAmxn
xnrn
rxnrxr
rmmxmxrnxnmxnmxm
V
V
UUVUA
2
11
21
00
0

[ ]
( ) ( )
[ ] H
AA
H
AA
H
AAH
A
H
A
AA
rmxrm
rxr
AAH
A
H
A
A
H
A UUUUUU
U
U
UU
I
I
UU
U
U
UU =+=





=





=





=
−−
2211
2
1
2121
2
1
0
0
[ ]
( ) ( )
[ ] H
AA
H
AA
H
AAH
A
H
A
AA
rnxrn
rxr
AAH
A
H
A
A
H
A VVVVVV
V
V
VV
I
I
VV
V
V
VV =+=





=





=








=
−−
2211
2
1
2121
2
1
0
0

65
SOLO Matrices
Let multiply by
and using:
mxpnxpmxn BXA =








H
H
U
U
2
1
we obtain: [ ]








=













Σ








BU
BU
X
V
V
UU
U
U
H
A
H
A
H
A
H
AA
I
AAH
A
H
A
m
2
1
2
11
21
2
1
00
0
  

or:








=













Σ
BU
BU
X
V
V
H
A
H
A
H
A
H
AA
2
1
2
11
00
0
or:
( ) ( )xprmmxp
H
A BU xmrm −=−
02
(m-r)xp - constraints equivalent
to condition Bmxp∈ℜ (Amxn)
mxp
H
Anxp
H
AA
BUXV rxmrxnrxr 111
=Σ
rxp - independent equations
nxp – unknowns
since r ≤ n → # Eq. ≤ # Unknown
( )H
AR
( )AN
( )H
AN
( )ARBXA =
11 yAx H
=
( )AdomX ∈
( )AcodomY ∈
Null Space of A
Ker (A)
span by VA2
H
Row Space of A
span by VA1
H
Column Space of A
span by UA1
span by UA2
B
( )
[ ] ( )
( ) ( ) ( ) ( ) 














Σ
=Σ=
−
−
−−−
−
H
A
H
A
rnxrmxrrm
rnrxA
AA
H
AAAmxn
xnrn
rxnrxr
rmmxmxrnxnmxnmxm
V
V
UUVUA
2
11
21
00
0


66
( ) ( )xprmmxp
H
A
BU xmrm −
=−
02
(m-r)xp - constraints equivalent
to condition Bmxp∈ℜ (Amxn)
SOLO Matrices
mxp
H
Anxp
H
AA
BUXV rxmrxnrxr 111
=Σ
rxp - independent equations
nxp – unknowns
since r ≤ n → # Eq. ≤ # Unknown
This equation is a Necessary and Sufficient Condition for any solutions of equation
Amxn Xnxp = Bmxp. It is equivalent to Bmxp∈ℜ (Amxn) or Bmxp ∩ N (AT
) = ∅.
If this condition is fulfilled, then from we have
nxp unknowns ≥ rxp independent equations, that means (n-r)xp degrees of freedom.
mxp
H
Anxp
H
AA BUXV rxmrxnrxr 111 =Σ
mxp
H
AAnxp
H
A BUXV rxmrxrrxn 1
1
11
−
Σ=
Since VA1
T
VA1=Ir & VA1
T
VA2 = 0 the
General Solution of Amxn Xnxp = Bmxp is:
( )
( ) ( )
( )
  
AN
xprnA
AR
mxp
H
AAAnxp
YVBUVX rnnx
T
rxmrxrnxr
∈
−
∈
−
−
+Σ= 21
1
11
where Y(n-r)xp is any (n-r)xp matrix, i.e. we
used all (n-r)xp degrees of freedom.
( )H
AR
( )AN
( )H
AN
( )ARBXA =
( )AdomX ∈
( )AcodomY ∈
Null Space of A
Ker (A)
span by VA2
H
Row Space of A
span by VA1
H
Column Space of A
span by UA1
span by UA2
BB has to be in the
column space of A
( ) ( ) =∩∈ ANBorARB

67
SOLO Matrices
Check:
( )
[ ] ( )
( ) ( ) ( ) ( )
( ) ( )( )
( )
[ ] ( )
( ) ( ) ( )
( ) ( )
( ) ( ) ( ) ( )
( )
[ ] ( )
( ) ( ) ( ) ( )
( )
[ ]
( )
( )
[ ] ( ) ( )
[ ]
( )
mxpmxp
xmrm
H
A
H
A
AA
mxpxmrm
H
A
mxp
H
A
AA
xprm
mxp
H
A
AA
xprn
mxp
H
AA
rnxrmxrrm
rnrxA
AA
xprn
I
A
H
Amxp
H
AAA
H
A
xprnA
H
Amxp
H
AA
I
A
H
A
rnxrmxrrm
rnrxA
AA
xprnAmxp
H
AAAH
A
H
A
rnxrmxrrm
rnrxA
AAnxp
H
AAAnxpmxn
BB
U
U
UUBU
BU
UU
BU
UU
Y
BU
UU
YVVBUVV
YVVBUVV
UU
YVBUV
V
V
UUXVUXA
rxm
rmmxmxr
rxm
rmmxmxr
rxm
rmmxmxr
rxmrxrrxr
rmmxmxr
rnnxxnrnrxmrxrnxrxnrn
rnnxrxnrxmrxrnxrrxnrxr
rmmxmxr
rnnxrxmrxrnxr
xnrn
rxnrxr
rmmxmxrnxnmxnmxm
=








=










=








=







Σ







Σ
=












+Σ
+Σ







Σ
=
+Σ















Σ
=Σ=
−
−
−−
−
−−−
−
−
−
−
−
−−−
−
−
−
−−−
−
−−
−−
−−−
−
−
−
−
−
2
1
21
0
2
1
21
1
21
1
1
11
21
221
1
1
0
12
0
211
1
1111
21
21
1
11
2
11
21
000
0
00
0
00
0









68
SOLO Matrices
where r is such that:
Algorithm to solve Amxn Xnxp = Bmxp:
(1) Compute s.v.d. of Amxn and partition according to:
( ) 0,,, 21211 >≥≥≥=Σ rrA diagrxr
σσσσσσ 
(2) Check if:
( ) ( )xprmmxp
H
A
BU xmrm −
=−
02
(3) If (2) is not true → no solution for (1)
( ) ( )

any
xprnAmxp
H
AAAnxp
YVBUVX rnnxrxmrxrnxr −
−
−
+Σ= 21
1
11
( )
[ ] ( )
( ) ( ) ( ) ( ) 














Σ
=Σ=
−
−
−−−
−
H
A
H
A
rnxrmxrrm
rnrxA
AA
H
AAAmxn
xnrn
rxnrxr
rmmxmxrnxnmxnmxm
V
V
UUVUA
2
11
21
00
0

If (2) is true → (n-r)xp solutions:

69
SOLO Matrices
Moore-Penrose Pseudoinverse of A:
( )
[ ] ( )
( ) ( ) ( ) ( )
H
AAAH
A
H
A
rmxrnxrrn
rmrxA
AnA
†
nxm rxmrxrnxr
xmrm
rxmrxr
rnnxmxr
UV
U
U
VVA 1
1
11
2
1
1
1
21
00
0
: −
−−−
−
−
Σ=















Σ
=
−
−

then
( )
[ ] ( )
( ) ( ) ( ) ( )
( )
[ ] ( )
( ) ( ) ( ) ( )
H
AAH
A
H
A
rmxrnxrrn
rmrxA
I
AAH
A
H
A
rnxrmxrrm
rnrxA
AA
†
mxn rxnnxr
xmrm
rxmrxr
nxn
rnnxnxr
xnrn
rxnrxr
rmmxmxrnxm
UU
U
U
VV
V
V
UUAA 11
2
1
1
1
21
2
11
21
00
0
00
0
=















Σ















Σ
=
−
−
−
−
−−−
−
−
−−−
−
  

( ) ( )
( ) ( ) ( )
H
AA
H
AA
H
AA
H
AA
†
nxmmxnmxm xmrmrmmxrxmmxrxmrmrmmxrxmmxr
UUUUUUUUAAI −−−−
=−+=− 22112211
also
( )
[ ] ( )
( ) ( ) ( ) ( )
( )
[ ] ( )
( ) ( ) ( ) ( )
H
AAH
A
H
A
rnxrmxrrm
rnrxA
I
AAH
A
H
A
rmxrnxrrn
rmrxA
AAnxm
†
mxn rxnnxr
xnrn
rxnrxr
mxm
rmmxmxr
xmrm
rxmrxr
rnnxnxr
VV
V
V
UU
U
U
VVAA 11
2
11
21
2
1
1
1
21
00
0
00
0
: =















Σ















Σ
=
−
−
−
−
−−−
−
−−−
−
−
  

( ) ( )
( ) ( ) ( )
H
AA
H
AA
H
AA
H
AAmxn
†
nxmnxn xnrnrmnxrxnnxrxnrnrmnxrxnnxr
VVVVVVVVAAI −−−−
=−+=− 22112211

70
SOLO Matrices
Moore-Penrose Pseudoinverse of A (continue - ):
Define also Znxp such that: ( ) ( ) nxp
H
Axprn
ZVY xrrn−
=− 2
:
( ) ( )

any
xprnAmxp
H
AAAnxp YVBUVX rnnxrxmrxrnxr −
−
−
+Σ= 21
1
11
Since, if
( ) ( )xprmmxp
H
A
BU xmrm −
=−
02
The solution of Amxn Xnxp = Bmxp is
( ) ( )
( ) ( ) ( )
H
AA
H
AA
H
AA
H
AAmxn
†
=−+=− 22112211
( )
[ ] ( )
( ) ( ) ( ) ( )
H
AAAH
A
H
A
rmxrnxrrn
rmrxA
AnA
†
nxm rxmrxrnxr
xmrm
rxmrxr
rnnxmxr
UV
U
U
VVA 1
1
11
2
1
1
1
21
00
0
: −
−−−
−
−
Σ=















Σ
=
−
−

Therefore: ( ) 
any
nxpmxn
†
nxmnxnmxp
†
nxmnxp
ZAAIBAX −+=
Note: By writing the solution this way we lose the fact that we have only (n-r)xp
different solutions as we have seen.
Check: ( ) ( )
( )

xprn
xnrnrnnxrxmrxrnxr
Y
nxp
H
AAmxp
H
AAAnxp
ZVVBUVX
−
−−
+Σ= −
221
1
11

71
SOLO Matrices
( ) ( ) ( ) 
any
nxpmxn
†
nxmnxnmxp
†
nxm
any
xprnAmxp
H
AAAnxp
ZAAIBAYVBUVX rnnxrxmrxrnxr
−+=+Σ= −
−
−
21
1
11
( )
( )
( )
( )
( ) ( ) ( )xprmmxp
H
AA
ANonBofprojection
mxp
H
BUU
BAAI
orANB
orARB
xmrmrmmx
T
−
+
==
−
Ο/=∩
∈
−−
0
0
22

  
Solutions exists iff:
( )H
AR
( )AN ( )H
AN
( )AR
BXA =
( ) YVZAAI A
†
nxn 2=−
BA†
Null Space of A
Ker (A)
span by VA2
H
Row Space of A
span by VA1
H Column Space of A
span by UA1
span by UA2
B
BA†Z
i.e. the projection (Imxm – Amxn Anxm
†
) of B on N (AH
) is zero.
( ) ( ) ( ) mxpmxp
H
AAmxp
†
nxmmxnmxm
BUUBAAI mnrmrmmx
0
0
22
==− −−

Table of Contents

72
SOLO Matrices
solutions always exist
( )H
AR
( )AN
( ) ( )ARBAN H
∈≡ &0
( )AR
BXA =
( ) YVZAAI A
†
nxn 2=−
BA†
Null Space of A
Ker (A)
span by VA2
H
Row Space of A
span by VA1
H Column Space of A
span by UA1
B
BA†Z
Since
( )[ ]
( ) 







Σ=Σ=
−
− H
A
H
A
mnmxAA
H
AAAmxn
xnmn
rxn
mxmmxmnxnmxnmxm
V
V
UVUA
2
1
1 0
( ) ( ) ( ) ( )xprmmxp
H
ArmmxA BUU xmrmrmmx −− ≡⇒≡ −−
00 22
nxp unknowns ≥ mxp equations, meaning (n-m)xp degrees of freedom
( ) ( )
( ) 
any
nxpmxn
†
nxmnxnmxp
†
nxm
any
xprnAmxp
H
AAAnxp
ZAAIBA
YVBUVX rnnxrxmrxrnxr
−+=
+Σ= −
−
−
21
1
11
( )
[ ]
( )
( ) 11
11
1
1
21
0
:
−−
−
−
=Σ=







Σ
= −
H
nxmmxn
H
nxm
H
AAA
H
A
xmmn
A
AnA
†
nxm
AAAUVUVVA mxmrxrnxrmxm
mxm
mnnxnxm

( )[ ]
( )
( )
[ ]
( )
mxm
H
A
xmmn
A
AAH
A
H
A
mnmxAA
†
nxmmxn IUVV
V
V
UAA mxm
mxm
mnnxnxm
xnmn
mxn
mxmmxm
=







Σ








Σ=
−
−
− −
−
0
0
1
1
21
2
1

Table of Contents

73
SOLO Matrices
Particular case (2) r = n ≤ n: mxp equations ≥ nxp unknowns, meaning (n-m)xp constraints
Only if solutions exist.
( ) ( )xprmmxp
H
A
BU xmrm −
=−
02
In this case we have nxp unknowns and mxp equations - (m-p)xp constraints =
nxp independent equations, i.e. a unique solution:
mxpnxmmxp
H
AAAnxp
BABUVX nxmnxnnxn
+−
=Σ= 1
1
11
( )
[ ]
( )
H
A
xnnm
A
AA
H
AAAmxn nxn
nxn
nmmxmxnnxnmxnmxm
VUUVUA







Σ
=Σ=
−
−
0
1
21

( )[ ]
( )
H
AAAH
A
H
A
nmnxAA
†
nxm nxmnxnnxn
xmnm
nxm
nxnnxn
UV
U
U
VA 1
1
2
11
0: −
−
−
Σ=








Σ=
−
( )[ ]
( )
( )
[ ]
( )
nxn
H
A
xnnm
A
AAH
A
H
A
nmnxAAmxn
†
nxm IVUU
U
U
VAA nxn
nxn
nmmxmxn
xmnm
nxm
nxnnxn
=







Σ








Σ=
−
−
−
−
−
0
0
1
21
2
11

( )H
AR
( ) 0=AN ( )H
AN
( )ARBXA =
BA†
1nx
y
Row Space of A
span by VA1
H
Column Space of A
span by UA1
span by UA2
BB has to be in the
column space of A
( ) ( ) =∩∈ ANBorARB
Table of Contents

74
SOLO Matrices
Y - pxm unknowns with pxn equations
pxnmxnpxm CAY =
where ( ) 0,,, 21211
>≥≥≥=Σ rrA
diagrxr
σσσσσσ 
[ ]
( ) ( )
[ ] H
AA
H
AA
H
AAH
A
H
A
AA
rmxrm
rxr
AAH
A
H
A
A
H
A UUUUUU
U
U
UU
I
I
UU
U
U
UU =+=





=





=





=
−−
2211
2
1
2121
2
1
0
0
[ ]
( ) ( )
[ ] H
AA
H
AA
H
AAH
A
H
A
AA
rnxrn
rxr
AAH
A
H
A
A
H
A VVVVVV
V
V
VV
I
I
VV
V
V
VV =+=





=





=








=
−−
2211
2
1
2121
2
1
0
0
( )H
AR
( )AN
( )H
AN
( )ARCAY =
11 yAx H
=
C
Y
Null Space of A
Ker (A)
span by VA2
H
Row Space of A
span by VA1
H
Column Space of A
span by UA1
span by UA2
( )
[ ] ( )
( ) ( ) ( ) ( ) 














Σ
=Σ=
−
−
−−−
−
H
A
H
A
rnxrmxrrm
rnrxA
AA
H
AAAmxn
xnrn
rxnrxr
rmmxmxrnxnmxnmxm
V
V
UUVUA
2
11
21
00
0


75
SOLO Matrices
we obtain: [ ] [ ] [ ]2121
2
11
21
00
0
AAAAH
A
H
AA
AA
VCVCVV
V
V
UUY  =













Σ
or:
( ) ( )rnpxApxn rnnx
VC −=−
02
px(n-r) - constraints equivalent
to condition Cpxn∈ℜ (Amxn
H
)
nxrrxrmxr ApxnAApxm
VCUY 111
=Σ
pxr - independent equations
pxm – unknowns
since r ≤ m → # Eq. ≤ # Unknown
( )
[ ] ( )
( ) ( ) ( ) ( ) 














Σ
=Σ=
−
−
−−−
−
H
A
H
A
rnxrmxrrm
rnrxA
AA
H
AAAmxn
xnrn
rxnrxr
rmmxmxrnxnmxnmxm
V
V
UUVUA
2
11
21
00
0

General Solution of YpxmAmxn = Cpxn ( )H
AR
( )AN
( )H
AN
( )ARCAY =
C
Y
Null Space of A
Ker (A)
span by VA2
T
Row Space of A
span by VA1
T
Column Space of A
span by UA1
span by UA2
C has to be in the
Row Space of A
H
AC
( )
( ) Ο/=∩
∈
ANC
orARC HLet post-multiply by
and using:
[ ]21
VVpxnmxnpxm CAY =
or: [ ] [ ]21
1
21
00
0
AA
A
AA
VCVCUUY  =




Σ

76
SOLO Matrices
Since UA1
T
UA1=Ir & UA1
T
UA2 = 0 the
General Solution of YpxmAmxn = Cpxn is:
where Xpx(m-r) is any px(m-r) matrix, i.e. we
used all px(m-r) degrees of freedom.
nxrrxrmxr ApxnAApxm
VCUY 111
=Σ
pxr - independent equations
pxm – unknowns
since r ≤ m → # Eq. ≤ # Unknown
This equation is a Necessary and Sufficient Condition for any solutions of equation
YpxmAmxn = Cpxn. It is equivalent to Cpxn∈ℜ (Amxn) or Cpxn ∩ N (AT
) = ∅.
If this condition is fulfilled, then from we have
nxp unknowns ≥ rxp independent equations, that means (n-r)xp degrees of freedom.
H
ApxnAApxm nxrrxrmxr
VCUY 111 =Σ
1
111
−
Σ= rxrnxrmxr AApxnApxm
VCUY
px(n-r) - constraints equivalent
to condition Cpxn∈ℜ (Amxn
H
)( ) ( )rnpxApxn rnnx
VC −=−
02
( ) ( )
H
A
any
rmpx
H
AAApxnpxm xmrmrxmrxrnxr
UXUVCY −−
−
+Σ= 21
1
11

( )H
AR
( )AN
( )H
AN
( )ARCAY =
C
Y
Null Space of A
Ker (A)
span by VA2
H
Row Space of A
span by VA1
H
Column Space of A
span by UA1
span by UA2
C has to be in the
Row Space of A
H
AC
( )
( ) Ο/=∩
∈
ANC
orARC H
H
AC

77
SOLO Matrices
Check:
( ) ( )
( ) ( )
[ ] ( )
( ) ( ) ( ) ( )
( ) ( ) ( ) ( ) ( ) ( )
( )
( ) ( ) ( ) ( )
( )[ ] ( )
( ) ( ) ( ) ( )
( )[ ]
( )
( )
( )
( )
( )
[ ]
( )
pxnH
A
H
A
AApxnH
A
H
A
ApxnApxn
H
A
H
A
rnpxApxnH
A
H
A
rnxrmxrrm
rnrxA
rmpxAApxn
H
A
H
A
rnxrmxrrm
rnrxA
I
A
H
ArmpxA
H
AAApxnA
H
Armpx
I
A
H
AAApxn
H
A
H
A
rnxrmxrrm
rnrxA
AA
H
Armpx
H
AAApxnmxnpxm
C
V
V
VVC
V
V
VCVC
V
V
VC
V
V
XVC
V
V
UUXUUVCUUXUUVC
V
V
UUUXUVCAY
xnrn
rxn
rnpxnxr
xnrn
rxn
rnpx
rnpxnxr
xnrn
rxn
nxr
xnrn
rxnrxr
rxrnxr
xnrn
rxnrxr
rmmxxmrmrmmxrxmrxrnxrmxrxmrmmxrrxmrxrnxr
xnrn
rxnrxr
rmmxmxrxmrmrxmrxrnxr
=








=


















=








=















Σ
Σ=















Σ








+Σ+Σ=















Σ
+Σ=
−
−
−
−
−
−−
−
−−−−
−
−−
−
−−−
−
−
−
−−−
−
−
−
−
−
−−−
−
−
−
2
1
21
2
1
0
21
2
1
1
2
111
11
2
11
22
0
21
1
11
0
1211
1
11
2
11
2121
1
11
0
00
0
00
0
00
0




  




78
SOLO Matrices
where r is such that:
Algorithm to solve YpxmAmxn = Cpxn:
(1) Compute s.v.d. of Amxn and partition according to:
( ) 0,,, 21211 >≥≥≥=Σ rrA diagrxr
σσσσσσ 
(2) Check if:
(3) If (2) is not true → no solution for (1)
( )
[ ] ( )
( ) ( ) ( ) ( ) 














Σ
=Σ=
−
−
−−−
−
H
A
H
A
rnxrmxrrm
rnrxA
AA
H
AAAmxn
xnrn
rxnrxr
rmmxmxrnxnmxnmxm
V
V
UUVUA
2
11
21
00
0

( ) ( )rnpxApxn rnnx
VC −=−
02
( ) ( )
H
A
any
rmpx
H
UXUVCY −−
−
+Σ= 21
1
11

If (2) is true → px(m-r) solutions:

79
SOLO Matrices
Define also Wpxm such that: ( ) ( )rmmxApxmrmpx UWX −
=− 2:
( ) ( )
( ) ( ) ( )
H
AA
H
AA
H
AA
H
AAmxn
†
=−+=− 22112211
( )
[ ] ( )
( ) ( ) ( ) ( )
H
AAAH
A
H
A
rmxrnxrrn
rmrxA
AnA
†
nxm rxmrxrnxr
xmrm
rxmrxr
rnnxmxr
UV
U
U
VVA 1
1
11
2
1
1
1
21
00
0
: −
−−−
−
−
Σ=















Σ
=
−
−

Therefore: 
( )†
nxmmxnmxm
any
pxm
†
nxmpxnpxm
AAIWACY −+=
Note: By writing the solution this way we lose the fact that we have only px(m-r)
different solutions as we have seen.
If ( ) ( )rnpxApxn rnnx
VC −
=−
02
the solution of YpxmAmxn = Cpxn is ( ) ( )
H
A
any
rmpx
H
UXUVCY −−
−
+Σ= 21
1
11

Check: ( )
( )
( )
H
A
X
Apxm
H
AAApxnpxm xmrm
rmpx
rmmxrxmrxrnxr
UUWUVCY −
−
−
+Σ= −
221
1
11


80
SOLO Matrices
Solutions exists iff:
i.e. the projection (Inxn – Anxm
†
Amxn) of C on N (A) is zero.
( ) ( )

( )†
nxmmxnmxm
any
pxm
†
nxmpxn
H
A
any
rmpx
H
AAApxnpxm
AAIWAC
UXUVCY xmrmrxmrxrnxr
−+=
+Σ= −−
−
21
1
11

( ) ( )
( )
( ) pxn
H
AApxnmxn
†
nxmnxnpxn xnrn
rnpx
rnnx
VVCAAIC 02
0
2 ==− −
−
−

( )H
AR
( )AN ( )H
AN
( )ARCAY =
C
pxm
Y
Null Space of A
Ker (A)
span by VA2
H
Row Space of A
span by VA1
H
Column Space of A
span by UA1
span by UA2
C has to be in the
Row Space of A
H
AC
( )
( ) orANC
orARC H
Ο/=∩
∈
( )
( )
( )
( )
( ) pxn
H
AApxn
ANonprojection
mxnnxmnxnpxn xnrn
rnpx
rnnx
VVCAAIC 02
0
2 ==− −
−
−
+
  
( )
( )
( ) ( )
H
Armpx
ANonWanyofprojection
nxmmxnmxmpxm xrrm
H
pxm
UXAAIW −−
+
=− 2
  
pxmW
H
AC
( )
( )
( )
( )
( )
( )
( ) pxn
H
AApxn
ANonprojection
mxn
†
nxmnxnpxn
H
xnrn
rnpx
rnnx
VVC
AAIC
orANC
orARC
02
0
2 ==
−
Ο/=∩
∈
−
−
−

  
†††
Table of Contents

81
SOLO Matrices
( )[ ]
( ) 







Σ=Σ=
−
− H
A
H
A
mnmxAA
H
AAAmxn
xnmn
rxn
mxmmxmnxnmxnmxm
V
V
UVUA
2
1
1 0
( )
[ ]
( )
( ) 11
11
1
1
21
0
:
−−
−
−
=Σ=







Σ
= −
H
nxmmxn
H
nxm
H
AAA
H
A
xmmn
A
AnA
†
nxm
AAAUVUVVA mxmrxrnxrmxm
mxm
mnnxnxm

( )[ ]
( )
( )
[ ]
( )
mxm
H
A
xmmn
A
AAH
A
H
A
mnmxAA
†
nxmmxn IUVV
V
V
UAA mxm
mxm
mnnxnxm
xnmn
mxn
mxmmxm
=







Σ








Σ=
−
−
− −
−
0
0
1
1
21
2
1

Only if solutions exist.( ) ( )rnpxApxn rnnx
VC −
=−
02
In this case we have pxm unknowns and pxn equations – px(n-m) constraints =
pxm independent equations, i.e. a unique solution:
†
nxmpxn
H
AAApxnpxm
AC
UVCY rxmrxrnxr
=
Σ= −
1
1
11
( )H
AR
( )AN
( ) 0=H
AN
( )ARCAY =
C
Null Space of A
Ker (A)
span by VA2
H
Row Space of A
span by VA1
H
Column Space of A
span by UA1
C has to be in the
Row Space of A
H
AC
( )
( )
( ) ( )
( )
( ) pxn
H
AApxnmxn
†
nxmnxnpxn
H
xnrn
rnpx
rnnx
VVCAAIC
orANC
orARC
02
0
2 ==−
Ο/=∩
∈
−
−
−

Table of Contents

82
SOLO Matrices
Particular case (2) r = n ≤ n: pxn equations ≥ pxm unknowns, meaning px(m-n) constraints
( )
[ ]
( )
H
A
xnnm
A
AA
H
AAAmxn nxn
nxn
nmmxmxnnxnmxnmxm
VUUVUA







Σ
=Σ=
−
−
0
1
21

( )[ ]
( )
H
AAAH
A
H
A
nmnxAA
†
nxm nxmnxnnxn
xmnm
nxm
nxnnxn
UV
U
U
VA 1
1
2
11
0: −
−
−
Σ=








Σ=
−
( )[ ]
( )
( )
[ ]
( )
nxn
H
A
xnnm
A
AAH
A
H
A
nmnxAAmxn
†
nxm
IVUU
U
U
VAA nxn
nxn
nmmxmxn
xmnm
nxm
nxnnxn
=







Σ








Σ=
−
−
−
−
−
0
0
1
21
2
11

Since solutions always exist( ) ( )rnpxApxn rnnx
VC −
≡−
02
pxm unknowns ≥ pxn equations, meaning px(m-n) degrees of freedom
( )H
AR
( ) 0=AN
( )H
AN
( )ARCAY =
C
Y
Row Space of A
span by VA1
H
Column Space of A
span by UA1
span by UA2
C has to be in the
Row Space of A
H
AC
( ) CAAC †
=
( ) H
A
UXAAIW 2
=− +
( )
( )
( ) ( )
( )
( ) pxn
H
AApxnmxn
†
nxmnxnpxn
H
xnrn
rnpx
rnnx
VVCAAIC
orANC
orARC
02
0
2
==−
Ο/=∩
∈
−
−
−

( ) ( )

( )†
nxmmxnmxm
any
pxm
†
nxmpxn
H
A
any
rmpx
H
AAApxnpxm
AAIWAC
UXUVCY xmrmrxmrxrnxr
−+=
+Σ= −−
−
21
1
11

Table of Contents

83
SOLO Matrices
( ) ( )
( ) ( ) 







−−−
−−−
=





−
×
−
×××
−
×
−
××××
−
×
−
×
−
××××
−
×
−
×
−
×××
−
××
××
11111
111111
mnnnnmmmnmmmmnnnnmmm
mnnnnmmmmnnnnmmmmnnn
mmnm
mnnn
BADCDCBADC
BADCBADCBA
CD
BA
if and exist.
1−
×nnA
1−
×nnC
Let find the inverse of such that:





××
××
mmnm
mnnn
PN
ML






××
××
mmnm
mnnn
CD
BA






=











××
××
××
××
××
××
mmnm
mnnn
mmnm
mnnn
mmnm
mnnn
I
I
PN
ML
CD
BA
0
0
Proof:
nnnmmnnnnn INBLA ××××× =+1 ( ) nnnnnmmmmnnn ILDCBA ×××
−
××× =−→
1
2 nmnmmmnnnm NCLD ××××× =+ 0
nnnmmmnm LDCN ××
−
×× −=→
1
3
mnmmmnmnnn PBMA ××××× =+ 0 mmmnnnmn PBAM ××
−
×× −=→
1
4
mmmmmmmnnm IPCMD ××××× =+ ( ) mmmmmnnnnmmm IPBADC ×××
−
××× =−→
1

84
SOLO Matrices
( ) ( )
( ) ( ) 







−−−
−−−
=





−
×
−
×××
−
×
−
××××
−
×
−
×
−
××××
−
×
−
×
−
×××
−
××
××
11111
111111
mmnm
mnnn
BADCDCBADC
BADCBADCBA
CD
BA
if and exist.
1−
×nnA
1−
×nnC
Let find the inverse of such that:





××
××
mmnm
mnnn
PN
ML






××
××
mmnm
mnnn
CD
BA






=











××
××
××
××
××
××
mmnm
mnnn
mmnm
mnnn
mmnm
mnnn
I
I
PN
ML
CD
BA
0
0
Proof (continue – 1):
1 ( ) nnnnnmmmmnnn
ILDCBA ×××
−
×××
=−
1
2 nnnmmmnm LDCN ××
−
×× −=
1
3
4
mmmnnnmn PBAM ××
−
×× −=
1
( ) mmmmmnnnnmmm IPBADC ×××
−
××× =−
1
( ) 11 −
×
−
×××× −= nmmmmnnnnn DCBAL
( ) 111 −
×
−
××××
−
×× −−= nmmmmnnnnmmmnm DCBADCN
( ) 11 −
×
−
×××× −= mnnnnmmmmm BADCP
( ) 111 −
×
−
××××
−
×× −−= mnnnnmmmmnnnmn BADCBAM
q.e.d.

85
SOLO Matrices






=











××
××
××
××
××
××
mmnm
mnnn
mmnm
mnnn
mmnm
mnnn
I
I
CD
BA
PN
ML
0
0
From:
( ) 1−
×××××
−=→ mmmnnmmmmm
CBNIPmmmmmmmnnm ICPBN ××××× =+we get:
( ) 11 −
×
−
×××× −= mnnnnmmmmm BADCPSubstitute:
and: ( ) 111 −
×
−
××××
−
×× −−= nmmmmnnnnmmmnm DCBADCN
( ) ( ) 1111111 −
××
−
×
−
××××
−
×
−
×
−
×
−
××× −+=− mmmnnmmmmnnnnmmmmmmnnnnmmm CBDCBADCCBADC
to obtain:
By inter-changing , in this identity, we obtain:nmmnnnmm DBAC ×××× ↔↔ ,
( ) ( ) 1111111 −
××
−
×
−
××××
−
×
−
×
−
×
−
×××
−+=− nnnmmnnnnmmmmnnnnnnmmmmnnn
ADBADCBAADCBA

86
SOLO
Pre-multiplying this identity by we obtain
Matrices
Let prove the identity:
( ) ( ) 111111 −
××
−
×
−
×××
−
×
−
××××
−
× −=− mmmnnmmmmnnnmnnnnmmmmnnn CBDCBABADCBA
( ) ( ) 1111 −
×
−
××××
−
××
−
××× −− mnnnnmmmmnnnnmmmmnnn BADCBADCBA
Proof:
( )( ) 1111 −
×
−
××××
−
××
−
××× −−≡ mnnnnmmmmnnnnmmmmnmn BADCBADCBB
( )( ) 11111 −
××
−
×
−
××××
−
×××
−
××
≡−−≡ mmmnmnnnnmmmmnnnnmmmmmmn
CBBADCBADCCB
( ) 11 −
×
−
××× − nmmmmnnn DCBA
( ) ( ) ( )
( ) 111
111111
−
××
−
×
−
×××
−
×
−
××××
−
××
−
×××
−
×
−
×××
−≡
−−−
mmmnnmmmmnnn
mnnnnmmmmnnnnmmmmnnnnmmmmnnn
CBDCBA
BADCBADCBADCBA
q.e.d.
( ) ( ) 111111 −
×
−
××××
−
×
−
××
−
×
−
×××
−=− nmmmmnnnnmmmnnnmmnnnnmmm
DCBADCADBADC
By inter-changing ,in the first identity, we obtain:nmmnnnmm DBAC ×××× ↔↔ ,
( ) ( ) 111111 −
××
−
×
−
×××
−
×
−
××××
−
× −=− nnnmmnnnnmmmnmmmmnnnnmmm ADBADCDCBADC
q.e.d.

87
SOLO Matrices
By using the identities:
( ) ( ) 111111 −
×
−
××××
−
×
−
××
−
×
−
××× −=− nmmmmnnnnmmmnnnmmnnnnmmm DCBADCADBADC
We obtain:
( ) ( )
( ) ( ) 







−−−
−−−
=





−
×
−
×××
−
×
−
××××
−
×
−
×
−
××××
−
×
−
×
−
×××
−
××
××
11111
111111
mmnm
mnnn
BADCDCBADC
BADCBADCBA
CD
BA
if and exist.
1−
×nnA
1−
×nnC
( ) ( )
( ) ( ) 







−−−
−−−+
=





−
×
−
×××
−
××
−
×
−
×××
−
×
−
××××
−
×
−
××
−
×
−
××××
−
×
−
×
−
××
××
11111
111111111
mnnnnmmmnnnmmnnnnmmm
mnnnnmmmmnnnnnnmmnnnnmmmmnnnnn
mmnm
mnnn
BADCADBADC
BADCBAADBADCBAA
CD
BA
( ) ( ) 1111111 −
××
−
×
−
××××
−
×
−
×
−
×
−
×××
−+=− nnnmmnnnnmmmmnnnnnnmmmmnnn
ADBADCBAADCBA

88
SOLO Matrices
If and exist, performing the computation M-1
M, we can prove:
1−
×nnA
1−
×nnC








−
=





−
×
−
××
−
×
×
−
×
−
××
××
111
11
00
mmnnnmmm
mnnn
mmnm
mnnn
CADC
A
CD
A
1
2







 −
=





−
××
−
××
−
×
−
×
−
××
××
1
1111
00 mmnm
mmmnnnnn
mmnm
mnnn
C
CBAA
C
BA
3








=





−
××
×
−
×
−
××
××
1
11
0
0
0
0
mmnm
mnnn
mmnm
mnnn
C
A
C
A
4 If and :
T
nnnn
AA ××
=
T
nnnn
CC ××
=
( ) ( )
( ) ( ) 







−−−
−−−
=





−
×
−
×××
−
×
−
××××
−
×
−
×
−
××××
−
×
−
×
−
×××
−
××
××
11111
111111
mnnnnm
T
mmnm
T
mmmnnnnm
T
mm
mnnnnm
T
mmmnnnnm
T
mmmnnn
mmnm
T
mnnn
BABCBCBABC
BABCBABCBA
CB
BA
Because this is a symmetric matrix
( ) ( ) 111111 −
××
−
×
−
×××
−
×
−
××××
−
× −=− nnnm
T
mnnnnm
T
mmnm
T
mmmnnnnm
T
mm ABBABCBCBABC
Also: ( ) ( ) 1111111 −
××
−
×
−
××××
−
×
−
×
−
×
−
××× −+=− mmmnnm
T
mmmnnnnm
T
mmmmmnnnnm
T
mm CBBCBABCCBABC

89
SOLO Matrices
If m=n and and also exist:
1−
×nnB
1−
×nn
D5
( ) ( )[ ] 111111 −
×
−
××
−
×××
−
×
−
××××
−
× −−=−− nnnnnnnnnnnnnnnnnnnnnnnn ABBADCBADCBA
( ) ( )[ ] 111111 −
×
−
××
−
×××
−
×
−
××××
−
× −−=−− nnnnnnnnnnnnnnnnnnnnnnnn CDDCBADCBADC
also
we obtain
( ) ( )
( ) ( ) 







−−
−−
=





−
×
−
×××
−
×
−
×××
−
×
−
×××
−
×
−
×××
−
××
××
1111
11111
nnnnnnnnnnnnnnnn
nnnnnnnnnnnnnnnn
nnnn
nnnn
BADCCDAB
ABCDDCBA
CD
BA
Table of Contents

90
SOLO Matrices
( ) ( ) 1111111 −
××
−
×
−
××
−
××
−
×
−
×
−
×××× +−=+ mmmnnmmmmnnnnmmmmmmnnnnmmm CBDCBADCCBADC1
Proof:
( ) ( ) 1111111 −
××
−
×
−
××××
−
×
−
×
−
×
−
××× −+=− mmmnnmmmmnnnnmmmmmmnnnnmmm CBDCBADCCBADC
In the identity:
substitute by .
1−
×nnA nnA ×−
Substitute by and by in (1).
1−
×nnAnnA × mmC ×
1−
×mmC2
( ) ( ) mmmnnmmmmnnnnmmmmmmnnnnmmm CBDCBADCCBADC ××
−
×××××××
−
×
−
××
−
× +−=+
1111
Substitute in (1) nnnn IA ×× =3
( ) ( ) 111111 −
××
−
×
−
××××
−
×
−
×
−
××× +−=+ mmmnnmmmmnnnnmmmmmmnnmmm CBDCBIDCCBDC
Substitute in (2) nnnn IA ×× =4
( ) ( ) mmmnnmmmmnnnnmmmmmmnnmmm CBDCBIDCCBDC ××
−
×××××××
−
××
−
× +−=+
111

91
SOLO Matrices
5Substitute in (1), (2), (3), (4) by . (We don’t assume symmetric and )nmD × nm
T
B × nnA × mmC ×
( ) ( ) 1111111 −
××
−
×
−
××
−
××
−
×
−
×
−
×××× +−=+ mmmnnm
T
mmmnnnnm
T
mmmmmnnnnm
T
mm CBBCBABCCBABC
( ) ( ) mmmnnm
T
mmmnnnnm
T
mmmmmnnnnm
T
mm CBBCBABCCBABC ××
−
×××××××
−
×
−
××
−
× +−=+
1111
( ) ( ) 111111 −
××
−
×
−
××××
−
×
−
×
−
××× +−=+ mmmnnm
T
mmmnnnnm
T
mmmmmnnm
T
mm CBBCBIBCCBBC
( ) ( ) mmmnnm
T
mmmnnnnm
T
mmmmmnnm
T
mm CBBCBIBCCBBC ××
−
×××××××
−
××
−
× +−=+
111
From this we get:
6Substitute in (3) mmmm IC ×× =
( ) ( ) mnnmmnnnnmmmmnnmmm BDBIDIBDI ×
−
×××××
−
××× +−=+
11
( ) ( )
( )( ) ( ) ( )
( ) ( ) ( ) ( ) mnnmmnnmmmmnnmmmmnnmmmmnnmmm
mnnmmmmnnmmnnmmmmnnmmmmnnmmm
mnnmmmmmmnnmmnnnnm
BDBDIBDIBDIBDI
BDIBDBDIBDIBDI
BDIIBDBID
××
−
×××
−
××××××
−
×××
−
×××××
−
×××
−
××××××
−
×××××
−
××××
+=+−++=
+=+−++=
+−=+
111
111
11

92
SOLO Matrices
7Substitute in the identity
( ) ( ) 111111 −
××
−
×
−
×××
−
×
−
××××
−
nnnn IA ×× −= and mmmm IC ×× = to obtain:
( ) ( ) mnnmmnnnmnnmmmmn BDBIBDIB ×
−
×××
−
×××× +=+
11
Pre-multiplying this by we get (6).nmD ×
By using a similar path with the identity
( ) ( ) 111111 −
××
−
×
−
×××
−
×
−
××××
−
nnnn IA ×× −= and mmmm IC ×× = to obtain:with
( ) ( ) nmmnnmmmnmmnnnnm DBDIDBID ×
−
×××
−
×××× +=+
11
Post-multiplying this by we get (6).mnB ×

93
SOLO Matrices
By matrix manipulation we obtain:
8In the identity
( ) ( ) 111111 −
××
−
×
−
×××
−
×
−
××××
−
pre-multiplying Anxn by and post-multiplying by Cmxm we get:
( ) ( ) mnnmmmmnnnnnmmmnnnnmmmmn BDCBAACBADCB ×
−
×
−
×××××
−
×
−
×××× −=−
1111
( ) ( ) mnnnnmmmmnnnmnnnnmmmmmmn BADCBIBADCIB ×
−−
××
−
×××
−
×
−
××
−
××× −=−
111111
Use now the identity
( ) ( ) 111111 −
××
−
×
−
×××
−
×
−
××××
−
Pre-multiplying by Cmxm and post-multiplying by Anxn we get:
( ) ( ) nmmnnnnmmmmmnnnmmmmnnnnm DBADCCADCBAD ×
−
×
−
×××××
−
×
−
×××× −=−
1111
By matrix manipulation we obtain
( ) ( ) nmmmmnnnnmmmnmmmmnnnnnnm DCBADIDCBAID ×
−−
××
−
×××
−
×
−
××
−
××× −=−
111111
Table of Contents

94
SOLO Matrices
Matrix Schwarz Inequality
( ) ( ) ( )QPPPQPQQ TTTTT 1−
≥
Table of Contents
Hermann Amandus
Schwarz
1843 - 1921
yxyx ≤>< ,
Let x, y be the elements of an Inner Product space X, than :
This is the Schwarz Inequality.
Let Pmxn and Qmxl be two matrices such that PT
P is nonsingular, then:
( ) ( ) ( ) CxxQPPPQPxxQQx TTTTTTT
∈∨≥
−1
i.e.,:
Furthermore equality holds if and only if exists a matrix Snxl such that Q = P S.
Proof:
Start from the inequality: and choose( ) ( ) 0≥−− SPQSPQ
T
( ) ( )QPPPS TT 1−
=
( ) ( )
( ) ( ) ( ) ( )
( ) ( ) ( ) 0
1
1111
≥−=
+−−=
+−−=−−
−
−−−−
QPPPQPQQ
QPPPPPPPPQQPPPPQQPPPPQQQ
SPPSQPSSPQQQSPQSPQ
TTTTT
TTTTTTTTTTTT
TTTTTTT
The inequality becomes equality if and only if : that is equivalent
with
( ) ( ) 0=−− SPQSPQ
T
SPQ =

95
SOLO Matrices
The trace of a square matrix is defined as ( ) ( )T
nn
n
i
iinn AtraceaAtrace ×
=
× == ∑1
:
q.e.d.
( ) ( )ABtraceBAtrace =1
Proof:
( ) ∑ ∑= =








=
n
i
n
j
jiij baBAtrace
1 1
( ) ( )BAtracebaabABtrace
n
i
n
j
jiij
n
j
n
i
ijji ==





= ∑∑∑ ∑ = == = 1 11 1
( ) ( )
( )
( )
( )
( ) ( )
( )
( )ABtraceBAtraceBAtraceABtraceABtraceBAtrace TTTT
111
=≠===2
Proof:
( ) ( ) ( )ABtraceBAtracebabaBAtrace
n
i
n
j
jiij
n
i
n
j
ijij
T
==







≠







= ∑ ∑∑ ∑ = == = 1 11 1
( ) ( )T
n
j
n
i
ijij
T
BAtraceabABtrace =





= ∑ ∑= =1 1
q.e.d.

96
SOLO Matrices
nn
n
i
=
× == ∑1
:
3
Proof:
q.e.d.
( ) ( ) ( )∑=
−
==
n
i
i APAPtraceAtrace
1
1
λ
where P is the eigenvector matrix of A related to the eigenvalue matrix Λ of A by










=Λ=
n
PPPA
λ
λ



0
01
( )
( )
( ) ( )AtraceAPPtracePAPtrace == −− 1
1
1










=Λ=
n
PPPA
λ
λ



0
01










=Λ=→ −
n
PAP
λ
λ



0
01
1
( ) ( ) ∑=
−
=Λ=→
n
i
itracePAPtace
1
1
λ

97
SOLO Matrices
nn
n
i
=
× == ∑1
:
Proof:
q.e.d.
Definition
4
( )AtraceA
ee =det
( )AtraceA
eeePe
P
PePPePe
n
i
i
======
∑=ΛΛΛ−Λ− 1
detdetdet
det
1
detdetdetdetdet 11
λ
If aij are the coefficients of the matrix Anxn and z is a scalar function of aij, i.e.:
( ) njiazz ij ,,1, ==
then is the matrix nxn whose coefficients i,j areA
z
∂
∂
nji
a
z
A
z
ijij
,,1,: =
∂
∂
=





∂
∂
(see Gelb “Applied Optimal Estimation”, pg.23)

98
SOLO Matrices
nn
n
i
=
× == ∑1
:
Proof:
q.e.d.
5
( ) ( ) ( )
A
Atrace
I
A
Atrace T
n
∂
∂
==
∂
∂ 1
( )



=
≠
==
∂
∂
=





∂
∂
∑= ji
ji
a
aA
Atrace
ij
n
i
ii
ijij
1
0
1
δ
6
( ) ( ) ( ) ( ) nmmnTTT
RBRCCBBC
A
BCAtrace
A
ABCtrace ××
∈∈==
∂
∂
=
∂
∂ 1
Proof:
( ) ( ) ( )[ ]ij
T
ji
m
p
pijp
ik
jl
n
l
m
p
n
k
klpklp
ijij
BCBCbcabc
aA
ABCtrace
===
∂
∂
=





∂
∂
∑∑∑∑ =
=
=
= = = 11 1 1
q.e.d.
7 If A, B, C ∈ Rnxn
,i.e. square matrices, then
( ) ( ) ( ) ( ) ( ) ( ) TTT
CBBC
A
BCAtrace
A
CABtrace
A
ABCtrace
==
∂
∂
=
∂
∂
=
∂
∂ 11

99
SOLO Matrices
nn
n
i
=
× == ∑1
:
Proof:
q.e.d.
8 ( )( ) ( )( ) ( )( )( )
nmmn
TTT
RBRCBC
A
ABCtrace
A
BCAtrace
A
ABCtrace ××
∈∈=
∂
∂
=
∂
∂
=
∂
∂ 721
9
( )( ) ( )( ) ( )( )
BC
A
BCAtrace
A
CABtrace
A
ABCtrace TTT 811
=
∂
∂
=
∂
∂
=
∂
∂
If A, B, C ∈ Rnxn
,i.e. square matrices, then
10
( ) T
A
A
Atrace
2
2
=
∂
∂
( ) ( ) ( )ij
T
jiji
n
l
n
m
mllm
ijijij
Aaaaa
aa
Atrace
A
Atrace
2
1 1
22
=+=





∂
∂
=
∂
∂
=





∂
∂
∑∑= =
11
( ) ( ) 1−
=
∂
∂ kT
k
Ak
A
Atrace
Proof:
( ) ( ) ( ) ( ) ( ) 1111 −−−−
=+++=
∂








⋅∂
=
∂
∂ kT
k
kTkTkT
k
k
AkAAA
A
AAAtrace
A
Atrace
  



q.e.d.

100
SOLO Matrices
nn
n
i
=
× == ∑1
:
Proof:
q.e.d.
12 ( ) T
A
A
e
A
etrace
=
∂
∂
( ) ( ) ( ) T
A
n
k
n
k
kT
n
kk
kT
n
n
k
k
n
n
k
k
n
A
eA
k
A
k
k
k
A
trace
Ak
A
trace
AA
etrace
===





∂
∂
=





∂
∂
=
∂
∂
∑ ∑∑∑ = =
→∞
→−
−
→∞
=
→∞
=
→∞
1 0
1
1
00 !
1
lim
!
lim
!
lim
!
lim
13
( )( ) ( )( ) ( )
( ) ( )( ) ( )( ) ( )
( ) ( )( ) ( )( ) ( )
TT
TTTTTTTTT
TTTTT
TTT
BACBAC
A
ACABtrace
A
BACAtrace
A
ABACtrace
A
CABAtrace
A
BACAtrace
A
CABAtrace
A
ACABtrace
A
BACAtrace
A
ABACtrace
+=
∂
∂
=
∂
∂
=
∂
∂
=
∂
∂
=
∂
∂
=
∂
∂
=
∂
∂
=
∂
∂
=
∂
∂
111
21
11
( ) ( ) ( ) ( ) ( )
( ) TTTT
TTT
BACBACCABBAC
A
ABACtrace
A
ABACtrace
A
ABACtrace
+=+==
∂
∂
+
∂
∂
=
∂
∂ + 86
2
2
1
1
Proof: q.e.d.
14
( ) ( )( )
A
A
AAtrace
A
AAtrace TT
2
13
=
∂
∂
=
∂
∂

101
SOLO Matrices
nn
n
i
=
× == ∑1
:
Proof:
15
( ) ( )TTTTT
ABBAABBA
A
ABAtrace
+=+=
∂
∂
Table of Contents
( ) ( ) ( )ij
TTTT
n
l
jlli
n
k
kijk
n
l
n
l
n
k
klmklm
ijijij
ABBAbababaa
aa
ABAtrace
A
ABAtrace
+=+=





∂
∂
=
∂
∂
=





∂
∂
∑∑∑∑∑ === = = 111 1 1
q.e.d.
16
( ) TTTTTT
CABBAC
A
ABACtrace
+=
∂
∂
( ) ( )
( )ij
TTTTTT
n
l
n
r
lirljr
n
k
n
m
mikmjk
n
l
n
r
rljrli
n
k
n
m
mikmjk
n
l
n
k
n
m
n
r
rlmrkmlk
ijijij
CABBACcabbac
abcbacabac
aa
ABACtrace
A
ABACtrace
+=+=
+=





∂
∂
=
∂
∂
=





∂
∂
∑∑∑∑
∑∑∑∑∑∑∑∑
= == =
= == == = = =
1 11 1
1 11 11 1 1 1
Proof:
q.e.d.

102
SOLO
References
[1] Pease, “Methods of Matrix Algebra” ,Mathematics in Science and Engineering,
Vol.16, Academic Press, 1965
Matrices
[2] S. Hermelin, “Robustness and Sensitivity Design of Linear Time-Invariant Systems”
PhD Thesis, Stanford University, 1986
Table of Contents

January 6, 2015 103
SOLO
Technion
Israeli Institute of Technology
1964 – 1968 BSc EE
1968 – 1971 MSc EE
Israeli Air Force
1970 – 1974
RAFAEL
Israeli Armament Development Authority
1974 – 2013
Stanford University
1983 – 1986 PhD AA
Matrices

104
SOLO
Derivatives of Matrices
Matrices
ljik
ij
kl
a
a
δδ=
∂
∂
For vector forms
j
i
ijii
i
i
y
x
y
x
y
x
y
x
y
x
y
x
∂
∂
=





∂
∂
∂
∂
=





∂
∂
∂
∂
=





∂
∂
:::
We have the following expressions:
( )
( )
( )
( ) ( ) ( )
( ) ( ) ( )
( ) ( )
( ) ( ) ( )
( )( ) ( )
( ) ( )
( ) ( )HH
TT
XX
XX
XXtraceX
XXtraceXX
XXXX
YXYXYX
YXYXYX
XtraceXtrace
YXYX
XX
constAifA
∂=∂
∂=∂
∂=∂
∂=∂
∂−=∂
∂⊗+⊗∂=⊗∂
∂⋅+⋅∂=⋅∂
∂=∂
∂+∂=+∂
=∂
==∂
−
−
−−−
1
1
111
detln
detdet
0
αα

105
SOLO
Derivatives of Determinants
Matrices
( ) ( ) 





∂
∂
=
∂
∂ −
x
Y
YtraceY
x
Y 1
det
det
( )
( )












∂
∂






∂
∂
−






∂
∂






∂
∂
+












∂
∂
∂
∂
=
∂
∂
∂
∂
−−
−−
−
x
Y
Y
x
Y
Ytrace
x
Y
Ytrace
x
Y
Ytrace
x
x
Y
YtraceY
x
x
Y
11
11
1
det
det
General Form

106
SOLO
Matrices
( ) ( )( )
( ) ( )
( ) ( )( ) ( )( ) 11
1
detdet
det
det
det
det
det
−−
−
==
∂
∂
=
∂
∂
=
∂
∂
∑
TT
ij
k
jk
ik
T
XBXAXBXA
X
BXA
XX
X
X
XX
X
X
δ
Linear Form
Square Forms
If X is Square and Invertible, then
( ) ( ) TT
T
XXAX
X
XAX −
=
∂
∂
det2
det
If X is Not Square but A is Symmetric, then
( ) ( ) ( ) 1
det2
det −
=
∂
∂
XAXXAXAX
X
XAX TT
T
If X is Not Square and A is Not Symmetric, then
( ) ( ) ( ) ( )[ ]11
det
det −−
+=
∂
∂
XAXXAXAXXAXAX
X
XAX TTTT
T

107
SOLO
Matrices
( ) ( )
( )
( )
( ) ( )
( ) ( ) Tk
k
T
T
T
T
T
XXk
X
X
XX
X
X
X
X
XX
X
X
XX
−
−
=
∂
∂
==
∂
∂
−=
∂
∂
=
∂
∂
det
det
22
detln
2
detln
2
detln
1T1-
†
†
Nonlinear Form

108
SOLO
Derivatives of an Inverse
Matrices
11
1
−−
−
∂
∂
−=
∂
∂
Y
x
Y
Y
x
Y
From this it follows
( ) ( ) ( )
( )
( ) ( )( )
( ) ( )
( )( ) ( ) ( )( )T
T
T
TTT
T
jlki
ij
kl
AXAX
X
AXtrace
XABX
X
BXAtrace
XX
X
X
XbaX
X
bXa
XX
X
X
11
1
11
1
11
1
1
11
1
det
det
−−
−
−−
−
−−
−
−−
−
−−
−
++−=
∂
+∂
−=
∂
∂
−=
∂
∂
−=
∂
∂
−=
∂
∂

109
SOLO
Derivatives of Matrices, Vectors and Scalar Forms
Matrices
First Order
( ) ( )
( )
( )
( ) ( )
( )
( ) ( )
( ) ( ) ( )ij
nm
mjin
mn
ij
T
ij
mn
njim
mn
ij
ij
ij
T
TTT
T
TT
T
T
TT
AJA
X
AX
AJA
X
AX
J
X
X
aa
X
aXa
X
aXa
ab
X
bXa
ba
X
bXa
a
x
xa
x
ax
==
∂
∂
==
∂
∂
=
∂
∂
=
∂
∂
=
∂
∂
=
∂
∂
=
∂
∂
=
∂
∂
=
∂
∂
δ
δ
n
mJ mn
↑
←
















=
000
010
000






110
SOLO
Matrices
Second Order
( ) ( )
( ) ( )[ ] ( ) ( )
( ) ( ) ( )
( ) ( ) jlikkl
ijijijT
ij
T
ilkjkl
T
lj
ij
kl
T
TTT
T
TT
TT
kl
kl
klmn
mnkl
ij
JXBJJBX
X
XBX
XBBX
X
XBX
bxBCDdxDCB
x
dxDCbxB
bccbX
X
cXXb
XXX
X
δδ
δδ
=+=
∂
∂
+=
∂
∂
+++=
∂
++∂
+=
∂
∂
=
∂
∂
∑∑ 2
n
mJ mn
↑
←
















=
000
010
000






111
SOLO
Matrices
Second Order (continue)
( ) ( )
( )
( ) ( )[ ] ( )( ) TTT
TTT
TT
T
T
bcbXDDcbXDcbX
X
bcXDcbXD
X
cXDXb
xBB
x
xBx
++=++
∂
∂
+=
∂
∂
+=
∂
∂
Assume W is symmetric
( ) ( )[ ] ( )
( ) ( )[ ] ( )
( ) ( )[ ] ( )
( ) ( )[ ] ( )
( ) ( )[ ] ( ) TT
T
T
T
TT
ssAxWsAxWsAx
A
sAxWsAxWsAx
x
sxWsxWsx
s
sxWsxWsx
x
sAxWAsAxWsAx
s
−−=−−
∂
∂
−=−−
∂
∂
−−=−−
∂
∂
−=−−
∂
∂
−−=−−
∂
∂
2
2
2
2
2

112
SOLO
Matrices
Higher Order and Nonlinear
( ) ( )∑
−
=
−−
=
∂
∂ 1
0
1
n
r
kl
rnijr
ij
kl
n
XJX
X
X
[ ] ( ) ( )
( )[ ] ( ) ( ) ( )[ ]∑
∑
−
=
−−−−
−−
−
=
+=
∂
∂
=
∂
∂
1
0
11
1
1
0
n
r
TrnTnTrrTnTrnnTnT
Trn
n
r
TTrnT
XbaXXXXbaXbXXa
X
XbaXbXa
X

Matrices ii

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Matrices ii

Similar to Matrices ii (20)

More from Solo Hermelin

More from Solo Hermelin (20)

Recently uploaded

Recently uploaded (20)

Matrices ii

Editor's Notes