Vectors and matrices are fundamental concepts in linear algebra. Vectors can be added, scaled, and combined linearly. Sets of vectors may be orthogonal, independent, or dependent. Matrices can represent linear transformations and be added, multiplied, inverted, and decomposed. Important matrix properties include rank, eigenvalues/eigenvectors, diagonalization, and decomposition.

1. Linear Algebra Review
1
Pattern Recognition

2. n-dimensional vector
• An n-dimensional vector v is denoted as
follows:
• The transpose vT is denoted as follows:

3. Inner (or dot) product
• Given vT = (x1, x2, . . . , xn) and wT = (y1, y2, . . . , yn),
their dot product defined as follows:
or
(scalar)

4. Orthogonal / Orthonormal vectors
• A set of vectors x1, x2, . . . , xn is orthogonal if
• A set of vectors x1, x2, . . . , xn is orthonormal if
k

5. Linear combinations
• A vector v is a linear combination of the
vectors v1, ..., vk if:
where c1, ..., ck are constants.
Example: any vector in R3 can be expressed
as a linear combinations of the unit vectors
i = (1, 0, 0), j = (0, 1, 0), and k = (0, 0, 1)
k
j
i

6. Space spanning
• A set of vectors S=(v1, v2, . . . , vk ) span some
space W if every vector v in W can be written
as a linear combination of the vectors in S
Example: the unit vectors i, j, and k span R3
k
j
i

7. Linear dependence
• A set of vectors v1, ..., vk are linearly dependent
if at least one of them (e.g., vj) can be written
as a linear combination of the rest:
(i.e., vj does not appear on the right side of the equation)
Example:

8. Linear independence
• A set of vectors v1, ..., vk is linearly independent
if no vector vj can be represented as a linear
combination of the remaining vectors, i.e. :
Example:
c1=c2=0

9. Vector basis
• A set of vectors v1, ..., vk forms a basis in
some vector space W if:
(1) (v1, ..., vk) span W
(2) (v1, ..., vk) are linearly independent
Some standard bases:
R2 R3 Rn

10. Orthogonal vector basis
• Basis vectors are not necessarily orthogonal.
• Any set of basis vectors (v1, ..., vk) can be
transformed to an orthogonal basis using the
Gram-Schmidt orthogonalization algorithm.
• Normalizing the basis vectors to “unit” length will
yield an orthonormal basis.
• More useful in practice since they simplify
calculations.

11. Vector Expansion/Projection
• Suppose v1, v2, . . . , vn is an orthogonal base
in W, then any v є W can be represented in
this basis as follows:
• The xi of the expansion can be computed as follows:
(vector expansion or projection)
(coefficients of expansion or projection)
where:
Note: if the basis is orthonormal, then vi.vi=1

12. Vector basis (cont’d)
• Given a set of basis vectors in W, each vector
in W can be represented (i.e., projected)
“uniquely” in this basis.
• How many vector bases are there in a vector
space?
– Many, e.g., translate/rotate a given vector basis to
obtain a new one!
– Some vector bases are preferred than others though.
– We will revisit this issue when we discuss Principal
Components Analysis (PCA).

13. Vector Reconstruction
• A vector x ϵ Rn is represented
by n components:
• Assuming the standard base
<v1, v2, …, vN> (i.e., unit vectors
in each dimension), each xi is
the projection of x along the
direction of vi:
• x can be “reconstructed” from
its projection coefficients as
follows:
13
1
2
.
.
:
.
.
.
N
x
x
x
 
 
 
 
 
 
 
 
 
 
 
 
 
x
T
T
i
i i
T
i i
v
x v
v v
 
x
x
1 1 2 2
1
...
N
i i N N
i
x v x v x v x v

    

x

14. Vector Reconstruction (cont’d)
• Example assuming n=2:
• Assuming the standard base
<v1=i, v2=j>, xi can be obtained
by projecting x along the
direction of vi:
• x can be reconstructed from its
projection coefficients as
follows:
14
1
2
3
:
4
x
x
   

   
 
 
x
 
1
1
3 4 3
0
T
x i
 
  
 
 
x
3 4
i j
 
x
 
2
0
3 4 4
1
T
x j
 
  
 
 
x
i
j

15. Matrix Operations
• Matrix addition/subtraction
– Add/Subtract corresponding elements.
– Matrices must be of same size.
• Matrix multiplication
Condition: n = q
m x n q x p m x p
n

16. Diagonal Matrices
Special case: Identity matrix

17. Matrix Transpose

18. Symmetric Matrices
Example:

19. Determinants
2 x 2
3 x 3
n x n
Properties:
(expanded along 1st column)
(expanded along kth column)

20. Matrix Inverse
• The inverse of a matrix A, denoted as A-1, has the
property:
A A-1 = A-1A = I
• A-1 exists only if
• Definitions
– Singular matrix: A-1 does not exist
– Ill-conditioned matrix: A is “close” to being singular

21. Matrix Inverse (cont’d)
• Useful properties:
• If A is orthogonal, then
A-1=AT

22. Matrix Trace
Properties:

23. Matrix Rank
• Defined as the size of the largest square sub-matrix
of A that has a non-zero determinant.
Example: has rank 3

24. Matrix Rank (cont’d)
• Alternatively, it can be defined as the maximum
number of linearly independent columns (or
rows) of A.
i.e., rank is not 4!
Example:

25. Matrix Rank (cont’d)
• Some useful properties:

26. Eigenvalues and Eigenvectors
• The vector v is an eigenvector of matrix A and
λ is an eigenvalue of A if:
Geometric interpretation: the linear transformation
implied by A cannot change the direction of the
eigenvectors v, only their magnitude.
(assuming v is non-zero)

27. Computing λ and v
• To compute the eigenvalues λ of a matrix A,
find the roots of the characteristic polynomial.
• The eigenvectors can then be computed:
Example:

28. Properties of λ and v
• Eigenvalues and eigenvectors are only
defined for square matrices.
• Eigenvectors are not unique (e.g., if v is an
eigenvector, so is kv) 
• Suppose λ1, λ2, ..., λn are the eigenvalues of
A, then:

29. Matrix diagonalization
• Given an n x n matrix A, find P such that:
P-1AP=Λ where Λ is diagonal
• Solution: set P = [v1 v2 . . . vn], where v1,v2 ,. . .
vn are the eigenvectors of A: eigenvalues of A
P-1AP=Λ

30. Matrix diagonalization (cont’d)
Example:
P-1AP=Λ

31. • An n x n matrix A is diagonalizable iff A has n
linearly independent eigenvectors.
– i.e., rank(P)=n, where P-1AP=Λ
• Theorem: If the eigenvalues of A are all distinct,
then the corresponding eigenvectors are
linearly independent (i.e., A is diagonalizable).
Are all n x n matrices
diagonalizable?

32. • If A is diagonalizable, then the corresponding
eigenvectors v1,v2 ,. . . vn form a basis in Rn
– Since v1,v2 ,. . . vn are linearly independent, they
also span Rn
• If A is also symmetric, its eigenvalues are
real, non-negative and the corresponding
eigenvectors are orthogonal.
– v1,v2 ,. . . vn form an orthogonal basis in Rn
Important Properties

33. Matrix decomposition
• If A is diagonalizable, then A can be
decomposed as follows:

34. Matrix decomposition (cont’d)
• If A is symmetric, matrix decomposition can
be simplified:
P-1=PT
A=PDPT=