This document provides a linear algebra review that covers topics such as vectors, matrices, eigenvalues, eigenvectors, linear independence, and the singular value decomposition. It defines key terms such as the dot product, transpose, inverse, determinant, rank, and orthogonality. Examples are provided to illustrate matrix operations and properties. The singular value decomposition is introduced as a way to write any matrix as the product of three matrices involving orthogonal matrices and a diagonal matrix of singular values.
The eigen values of a Hermitian matrix are always real. This is because for a Hermitian matrix A, the quadratic form x*Ax is always real for any vector x. Now, if λ is an eigen value of A corresponding to the eigenvector v, then we have:
λv*v = v*Av
λv*v = v*λv (since Av = λv)
λv*v = λv*v
Therefore, λ must be real. Similarly, for a real symmetric matrix, the quadratic form x'Ax is always real. Hence, the eigen values must be real.
So in summary, the eigen values of both Hermitian and real
The document discusses the multiple linear regression model and ordinary least squares (OLS) estimation. It presents the econometric model, where a dependent variable is modeled as a linear function of explanatory variables, plus an error term. It describes the assumptions of the linear regression model, including linearity, independence of observations, exogeneity of regressors, and properties of the error term. It then discusses OLS estimation, goodness of fit, hypothesis testing, confidence intervals, and asymptotic properties of the OLS estimator.
Scatter diagrams and correlation and simple linear regresssionAnkit Katiyar
The document discusses scatter diagrams, correlation, and linear regression. It defines key terms like predictor and response variables, positively and negatively associated variables, and the correlation coefficient. It also describes how to calculate the linear correlation coefficient and interpret it. The document shows an example of using least squares regression to fit a line to productivity and experience data. It provides formulas to calculate the slope and intercept of the regression line and how to make predictions with the line. However, predictions should stay within the scope of the observed data used to fit the model.
The document discusses operators, conditional constructs, and looping constructs in C#. It covers arithmetic, assignment, unary, comparison and logical operators. It also covers if/else statements, switch/case statements, while loops, do-while loops, for loops, and the break and continue statements for controlling program flow. The goal is to teach users how to use these programming elements to process data, make conditional decisions, and repeat code blocks.
JavaScript is used to add interactivity to web pages. It can validate forms, detect browsers, create cookies, and more. JavaScript is the most popular scripting language on the internet and works in all major browsers. JavaScript is an interpreted language that is usually embedded directly into HTML pages. It is commonly used to dynamically update content and validate user input. Java and JavaScript are two completely different languages - Java is more complex and powerful while JavaScript is lightweight.
The document discusses measures of dispersion such as variance, standard deviation, and the coefficient of variation. It defines variance as the average squared deviation from the mean and standard deviation as the positive square root of the variance. The coefficient of variation measures relative dispersion by dividing the standard deviation by the mean. It is unit-free and allows for comparison across distributions. The document also covers Chebyshev's inequality and how it relates to the proportion of data within a given number of standard deviations from the mean.
This document outlines key concepts in linear models and estimation that will be covered in the STA721 Linear Models course, including:
1) Linear regression models decompose observed data into fixed and random components.
2) Maximum likelihood estimation finds parameter values that maximize the likelihood function.
3) Linear restrictions on the mean vector μ define a subspace and equivalent parameterizations represent the same subspace.
4) Inference should be independent of the parameterization or coordinate system used to represent μ.
This document discusses differential equations. It defines differential equations as equations relating an unknown function and one or more of its derivatives. It describes the order and degree of differential equations. Several examples of first order differential equations are given and solved using techniques like direct integration, variable separation, and solving homogeneous equations. The key steps for solving first order differential equations are outlined.
The eigen values of a Hermitian matrix are always real. This is because for a Hermitian matrix A, the quadratic form x*Ax is always real for any vector x. Now, if λ is an eigen value of A corresponding to the eigenvector v, then we have:
λv*v = v*Av
λv*v = v*λv (since Av = λv)
λv*v = λv*v
Therefore, λ must be real. Similarly, for a real symmetric matrix, the quadratic form x'Ax is always real. Hence, the eigen values must be real.
So in summary, the eigen values of both Hermitian and real
The document discusses the multiple linear regression model and ordinary least squares (OLS) estimation. It presents the econometric model, where a dependent variable is modeled as a linear function of explanatory variables, plus an error term. It describes the assumptions of the linear regression model, including linearity, independence of observations, exogeneity of regressors, and properties of the error term. It then discusses OLS estimation, goodness of fit, hypothesis testing, confidence intervals, and asymptotic properties of the OLS estimator.
Scatter diagrams and correlation and simple linear regresssionAnkit Katiyar
The document discusses scatter diagrams, correlation, and linear regression. It defines key terms like predictor and response variables, positively and negatively associated variables, and the correlation coefficient. It also describes how to calculate the linear correlation coefficient and interpret it. The document shows an example of using least squares regression to fit a line to productivity and experience data. It provides formulas to calculate the slope and intercept of the regression line and how to make predictions with the line. However, predictions should stay within the scope of the observed data used to fit the model.
The document discusses operators, conditional constructs, and looping constructs in C#. It covers arithmetic, assignment, unary, comparison and logical operators. It also covers if/else statements, switch/case statements, while loops, do-while loops, for loops, and the break and continue statements for controlling program flow. The goal is to teach users how to use these programming elements to process data, make conditional decisions, and repeat code blocks.
JavaScript is used to add interactivity to web pages. It can validate forms, detect browsers, create cookies, and more. JavaScript is the most popular scripting language on the internet and works in all major browsers. JavaScript is an interpreted language that is usually embedded directly into HTML pages. It is commonly used to dynamically update content and validate user input. Java and JavaScript are two completely different languages - Java is more complex and powerful while JavaScript is lightweight.
The document discusses measures of dispersion such as variance, standard deviation, and the coefficient of variation. It defines variance as the average squared deviation from the mean and standard deviation as the positive square root of the variance. The coefficient of variation measures relative dispersion by dividing the standard deviation by the mean. It is unit-free and allows for comparison across distributions. The document also covers Chebyshev's inequality and how it relates to the proportion of data within a given number of standard deviations from the mean.
This document outlines key concepts in linear models and estimation that will be covered in the STA721 Linear Models course, including:
1) Linear regression models decompose observed data into fixed and random components.
2) Maximum likelihood estimation finds parameter values that maximize the likelihood function.
3) Linear restrictions on the mean vector μ define a subspace and equivalent parameterizations represent the same subspace.
4) Inference should be independent of the parameterization or coordinate system used to represent μ.
This document discusses differential equations. It defines differential equations as equations relating an unknown function and one or more of its derivatives. It describes the order and degree of differential equations. Several examples of first order differential equations are given and solved using techniques like direct integration, variable separation, and solving homogeneous equations. The key steps for solving first order differential equations are outlined.
The document discusses the different types of variables in Ruby including global variables, instance variables, class variables, local variables, constants, pseudo-variables, and literals. It provides examples of how to declare and use each variable type in Ruby code. It also covers the basic data types like integers, floats, strings, arrays, hashes, and ranges that can be used as literals in Ruby.
The document discusses pseudospectra as an alternative to eigenvalues for analyzing non-normal matrices and operators. It defines three equivalent definitions of pseudospectra: (1) the set of points where the resolvent is larger than ε-1, (2) the set of points that are eigenvalues of a perturbed matrix with perturbation smaller than ε, and (3) the set of points where the resolvent applied to a unit vector is larger than ε. It also shows that pseudospectra are nested sets and their intersection is the spectrum. The definitions extend to operators on Hilbert spaces using singular values.
Here are the key steps to find the eigenvalues of the given matrix:
1) Write the characteristic equation: det(A - λI) = 0
2) Expand the determinant: (1-λ)(-2-λ) - 4 = 0
3) Simplify and factor: λ(λ + 1)(λ + 2) = 0
4) Find the roots: λ1 = 0, λ2 = -1, λ3 = -2
Therefore, the eigenvalues of the given matrix are -1 and -2.
This document discusses techniques for semi-local string comparison, which involves comparing whole strings to substrings or prefixes to suffixes. It introduces implicit unit-Monge matrices, which allow efficient querying of string distances. Several algorithms are described that use these matrices, including the seaweed method and transposition network method.
4. Linear Algebra for Machine Learning: Eigenvalues, Eigenvectors and Diagona...Ceni Babaoglu, PhD
The seminar series will focus on the mathematical background needed for machine learning. The first set of the seminars will be on "Linear Algebra for Machine Learning". Here are the slides of the fourth part which is discussing eigenvalues, eigenvectors and diagonalization.
Here is the link of the first part which was discussing linear systems: https://www.slideshare.net/CeniBabaogluPhDinMat/linear-algebra-for-machine-learning-linear-systems/1
Here are the slides of the second part which was discussing basis and dimension:
https://www.slideshare.net/CeniBabaogluPhDinMat/2-linear-algebra-for-machine-learning-basis-and-dimension
Here are the slides of the third part which is discussing factorization and linear transformations.
https://www.slideshare.net/CeniBabaogluPhDinMat/3-linear-algebra-for-machine-learning-factorization-and-linear-transformations-130813437
My talk on "State Space C-Reductions for Concurrent Systems in Rewriting Logic" held at the International ETAPS Workshop on Graph Inspection and Traversal Engineering (GRAPHITE 2013).
Full manuscript available here: http://eprints.imtlucca.it/1350/
State Space c-Reductions of Concurrent Systems in Rewriting Logic @ ETAPS Wor...Alberto Lluch Lafuente
We present c-reductions, a state space reduction technique. The rough idea is to exploit some equivalence relation on states (possibly capturing system regularities) that preserves behavioral properties, and explore the induced quotient system. This is done by means of a canonizer function, which maps each state into one (of the) canonical representative(s) of its equivalence class. The approach exploits the expressiveness of rewriting logic and its realization in Maude to enjoy several advantages over similar approaches: flexibility and simplicity in the definition of the reductions (supporting not only traditional symmetry reductions, but also name reuse and name abstraction); reasoning support for checking and proving correctness of the reductions; and automatization of the reduction infrastructure via Maude's meta-programming features. The approach has been validated over a set of representative case studies, exhibiting comparable results with respect to other tools.
This document discusses using the discriminant of a quadratic equation to determine the number and type of roots. It defines the discriminant as b^2 - 4ac and shows how its value relates to whether the roots are real or complex, rational or irrational. Examples are worked through, showing how to find the discriminant, roots, and sketch the graph. The key findings are summarized in a chart relating the value of the discriminant to the type and number of roots. Readers are then asked to find roots, sketch graphs, and evaluate discriminants for additional practice problems.
IVS-B UNIT-1_merged. Semester 2 fundamental of sciencepdf42Rnu
Unit-1 covers topics related to error analysis, graphing, and logarithms. It discusses types of errors, propagation of errors through addition, subtraction, multiplication, division, and powers. It also defines standard deviation and provides examples of calculating it. Graphing concepts like dependent and independent variables, linear and nonlinear functions, and plotting graphs from equations are explained. Logarithm rules and properties are also introduced.
This document provides information on shell operators in 3 sections:
1) It discusses arithmetic operators and how to perform arithmetic evaluations in the shell using expr and $((expression)).
2) It covers relational operators for numeric comparisons and conditional statements.
3) It describes string, file test, and boolean operators for evaluating strings and conditions.
The document defines rational and irrational numbers. Rational numbers can be written as fractions with integer numerators and denominators. Irrational numbers are real numbers that cannot be written as fractions, such as the square roots of non-perfect squares. The document provides examples of evaluating, approximating, and simplifying square roots of numbers using properties of perfect squares. It also discusses using calculators to evaluate square roots.
Lecture 9: Dimensionality Reduction, Singular Value Decomposition (SVD), Principal Component Analysis (PCA). (ppt,pdf)
Appendices A, B from the book “Introduction to Data Mining” by Tan, Steinbach, Kumar.
The document provides information about a test for candidates applying for an M.Tech in Computer Science. It describes:
1) The test will have two parts - a morning objective test (Test MIII) and an afternoon short answer test (Test CS).
2) The CS test booklet will have two groups - Group A covering analytical ability and mathematics at the B.Sc. pass level, and Group B covering advanced topics in mathematics, statistics, physics, computer science, and engineering at the B.Sc. Hons. and B.Tech. levels.
3) Sample questions are provided for both Group A (mathematical reasoning and basic concepts) and Group B (advanced topics in real analysis
The document describes a test for candidates applying for an M.Tech. in Computer Science. [The test consists of two parts - an objective test in the morning and a short answer test in the afternoon. The short answer test has two groups - Group A covers analytical ability and mathematics at the B.Sc. level, while Group B covers additional topics in mathematics, statistics, physics, computer science, or engineering depending on the candidate's choice.] The document provides sample questions testing concepts in mathematics including algebra, calculus, number theory, and logic.
Quantitative Methods for Lawyers - Class #21 - Regression Analysis - Part 4Daniel Katz
This document summarizes a class on regression analysis in R. It discusses using the Stargazer package to generate LaTeX code for regression tables. It covers interpreting regression output, including thinking about relationships between variables in a ceteris paribus manner and discussing how changes in independent variables affect the dependent variable. The document also discusses checking for non-linear relationships and adding transformed or interaction variables to improve model fit.
Dimensionality reduction techniques like principal component analysis (PCA) and singular value decomposition (SVD) are important for analyzing high-dimensional data by finding patterns in the data and expressing the data in a lower-dimensional space. PCA and SVD decompose a data matrix into orthogonal principal components/singular vectors that capture the maximum variance in the data, allowing the data to be represented in fewer dimensions without losing much information. Dimensionality reduction is useful for visualization, removing noise, discovering hidden correlations, and more efficiently storing and processing the data.
- Dimensionality reduction techniques assign instances to vectors in a lower-dimensional space while approximately preserving similarity relationships. Principal component analysis (PCA) is a common linear dimensionality reduction technique.
- Kernel PCA performs PCA in a higher-dimensional feature space implicitly defined by a kernel function. This allows PCA to find nonlinear structure in data. Kernel PCA computes the principal components by finding the eigenvectors of the normalized kernel matrix.
- For a new data point, its representation in the lower-dimensional space is given by projecting it onto the principal components in feature space using the kernel trick, without explicitly computing features.
1) A matrix is a rectangular array of numbers arranged in rows and columns. The dimensions of a matrix are specified by the number of rows and columns.
2) The inverse of a square matrix A exists if and only if the determinant of A is not equal to 0. The inverse of A, denoted A^-1, is the matrix that satisfies AA^-1 = A^-1A = I, where I is the identity matrix.
3) For two matrices A and B to be inverses, their product must result in the identity matrix regardless of order, i.e. AB = BA = I. This shows that one matrix undoes the effect of the other.
Dimensionality reduction techniques like random projection and latent semantic indexing (LSI) can reduce the number of dimensions in a document vector space while approximately preserving distances. Random projection selects random orthogonal projection axes to map vectors to a lower-dimensional space. LSI uses singular value decomposition to identify related terms and documents, projecting the vector space to its principal components to group semantically similar words. LSI provides a better approximation than random projection by exploiting relationships in the data. Dimensionality reduction speeds up information retrieval tasks like computing document similarities and ranking query results.
The document discusses the different types of variables in Ruby including global variables, instance variables, class variables, local variables, constants, pseudo-variables, and literals. It provides examples of how to declare and use each variable type in Ruby code. It also covers the basic data types like integers, floats, strings, arrays, hashes, and ranges that can be used as literals in Ruby.
The document discusses pseudospectra as an alternative to eigenvalues for analyzing non-normal matrices and operators. It defines three equivalent definitions of pseudospectra: (1) the set of points where the resolvent is larger than ε-1, (2) the set of points that are eigenvalues of a perturbed matrix with perturbation smaller than ε, and (3) the set of points where the resolvent applied to a unit vector is larger than ε. It also shows that pseudospectra are nested sets and their intersection is the spectrum. The definitions extend to operators on Hilbert spaces using singular values.
Here are the key steps to find the eigenvalues of the given matrix:
1) Write the characteristic equation: det(A - λI) = 0
2) Expand the determinant: (1-λ)(-2-λ) - 4 = 0
3) Simplify and factor: λ(λ + 1)(λ + 2) = 0
4) Find the roots: λ1 = 0, λ2 = -1, λ3 = -2
Therefore, the eigenvalues of the given matrix are -1 and -2.
This document discusses techniques for semi-local string comparison, which involves comparing whole strings to substrings or prefixes to suffixes. It introduces implicit unit-Monge matrices, which allow efficient querying of string distances. Several algorithms are described that use these matrices, including the seaweed method and transposition network method.
4. Linear Algebra for Machine Learning: Eigenvalues, Eigenvectors and Diagona...Ceni Babaoglu, PhD
The seminar series will focus on the mathematical background needed for machine learning. The first set of the seminars will be on "Linear Algebra for Machine Learning". Here are the slides of the fourth part which is discussing eigenvalues, eigenvectors and diagonalization.
Here is the link of the first part which was discussing linear systems: https://www.slideshare.net/CeniBabaogluPhDinMat/linear-algebra-for-machine-learning-linear-systems/1
Here are the slides of the second part which was discussing basis and dimension:
https://www.slideshare.net/CeniBabaogluPhDinMat/2-linear-algebra-for-machine-learning-basis-and-dimension
Here are the slides of the third part which is discussing factorization and linear transformations.
https://www.slideshare.net/CeniBabaogluPhDinMat/3-linear-algebra-for-machine-learning-factorization-and-linear-transformations-130813437
My talk on "State Space C-Reductions for Concurrent Systems in Rewriting Logic" held at the International ETAPS Workshop on Graph Inspection and Traversal Engineering (GRAPHITE 2013).
Full manuscript available here: http://eprints.imtlucca.it/1350/
State Space c-Reductions of Concurrent Systems in Rewriting Logic @ ETAPS Wor...Alberto Lluch Lafuente
We present c-reductions, a state space reduction technique. The rough idea is to exploit some equivalence relation on states (possibly capturing system regularities) that preserves behavioral properties, and explore the induced quotient system. This is done by means of a canonizer function, which maps each state into one (of the) canonical representative(s) of its equivalence class. The approach exploits the expressiveness of rewriting logic and its realization in Maude to enjoy several advantages over similar approaches: flexibility and simplicity in the definition of the reductions (supporting not only traditional symmetry reductions, but also name reuse and name abstraction); reasoning support for checking and proving correctness of the reductions; and automatization of the reduction infrastructure via Maude's meta-programming features. The approach has been validated over a set of representative case studies, exhibiting comparable results with respect to other tools.
This document discusses using the discriminant of a quadratic equation to determine the number and type of roots. It defines the discriminant as b^2 - 4ac and shows how its value relates to whether the roots are real or complex, rational or irrational. Examples are worked through, showing how to find the discriminant, roots, and sketch the graph. The key findings are summarized in a chart relating the value of the discriminant to the type and number of roots. Readers are then asked to find roots, sketch graphs, and evaluate discriminants for additional practice problems.
IVS-B UNIT-1_merged. Semester 2 fundamental of sciencepdf42Rnu
Unit-1 covers topics related to error analysis, graphing, and logarithms. It discusses types of errors, propagation of errors through addition, subtraction, multiplication, division, and powers. It also defines standard deviation and provides examples of calculating it. Graphing concepts like dependent and independent variables, linear and nonlinear functions, and plotting graphs from equations are explained. Logarithm rules and properties are also introduced.
This document provides information on shell operators in 3 sections:
1) It discusses arithmetic operators and how to perform arithmetic evaluations in the shell using expr and $((expression)).
2) It covers relational operators for numeric comparisons and conditional statements.
3) It describes string, file test, and boolean operators for evaluating strings and conditions.
The document defines rational and irrational numbers. Rational numbers can be written as fractions with integer numerators and denominators. Irrational numbers are real numbers that cannot be written as fractions, such as the square roots of non-perfect squares. The document provides examples of evaluating, approximating, and simplifying square roots of numbers using properties of perfect squares. It also discusses using calculators to evaluate square roots.
Lecture 9: Dimensionality Reduction, Singular Value Decomposition (SVD), Principal Component Analysis (PCA). (ppt,pdf)
Appendices A, B from the book “Introduction to Data Mining” by Tan, Steinbach, Kumar.
The document provides information about a test for candidates applying for an M.Tech in Computer Science. It describes:
1) The test will have two parts - a morning objective test (Test MIII) and an afternoon short answer test (Test CS).
2) The CS test booklet will have two groups - Group A covering analytical ability and mathematics at the B.Sc. pass level, and Group B covering advanced topics in mathematics, statistics, physics, computer science, and engineering at the B.Sc. Hons. and B.Tech. levels.
3) Sample questions are provided for both Group A (mathematical reasoning and basic concepts) and Group B (advanced topics in real analysis
The document describes a test for candidates applying for an M.Tech. in Computer Science. [The test consists of two parts - an objective test in the morning and a short answer test in the afternoon. The short answer test has two groups - Group A covers analytical ability and mathematics at the B.Sc. level, while Group B covers additional topics in mathematics, statistics, physics, computer science, or engineering depending on the candidate's choice.] The document provides sample questions testing concepts in mathematics including algebra, calculus, number theory, and logic.
Quantitative Methods for Lawyers - Class #21 - Regression Analysis - Part 4Daniel Katz
This document summarizes a class on regression analysis in R. It discusses using the Stargazer package to generate LaTeX code for regression tables. It covers interpreting regression output, including thinking about relationships between variables in a ceteris paribus manner and discussing how changes in independent variables affect the dependent variable. The document also discusses checking for non-linear relationships and adding transformed or interaction variables to improve model fit.
Dimensionality reduction techniques like principal component analysis (PCA) and singular value decomposition (SVD) are important for analyzing high-dimensional data by finding patterns in the data and expressing the data in a lower-dimensional space. PCA and SVD decompose a data matrix into orthogonal principal components/singular vectors that capture the maximum variance in the data, allowing the data to be represented in fewer dimensions without losing much information. Dimensionality reduction is useful for visualization, removing noise, discovering hidden correlations, and more efficiently storing and processing the data.
- Dimensionality reduction techniques assign instances to vectors in a lower-dimensional space while approximately preserving similarity relationships. Principal component analysis (PCA) is a common linear dimensionality reduction technique.
- Kernel PCA performs PCA in a higher-dimensional feature space implicitly defined by a kernel function. This allows PCA to find nonlinear structure in data. Kernel PCA computes the principal components by finding the eigenvectors of the normalized kernel matrix.
- For a new data point, its representation in the lower-dimensional space is given by projecting it onto the principal components in feature space using the kernel trick, without explicitly computing features.
1) A matrix is a rectangular array of numbers arranged in rows and columns. The dimensions of a matrix are specified by the number of rows and columns.
2) The inverse of a square matrix A exists if and only if the determinant of A is not equal to 0. The inverse of A, denoted A^-1, is the matrix that satisfies AA^-1 = A^-1A = I, where I is the identity matrix.
3) For two matrices A and B to be inverses, their product must result in the identity matrix regardless of order, i.e. AB = BA = I. This shows that one matrix undoes the effect of the other.
Dimensionality reduction techniques like random projection and latent semantic indexing (LSI) can reduce the number of dimensions in a document vector space while approximately preserving distances. Random projection selects random orthogonal projection axes to map vectors to a lower-dimensional space. LSI uses singular value decomposition to identify related terms and documents, projecting the vector space to its principal components to group semantically similar words. LSI provides a better approximation than random projection by exploiting relationships in the data. Dimensionality reduction speeds up information retrieval tasks like computing document similarities and ranking query results.
1. Linear Algebra Review
By Tim K. Marks
UCSD
Borrows heavily from:
Jana Kosecka kosecka@cs.gmu.edu
http://cs.gmu.edu/~kosecka/cs682.html
Virginia de Sa
Cogsci 108F Linear Algebra review
UCSD
Vectors
The length of x, a.k.a. the norm or 2-norm of x, is
x = x12 + x 2 + L + x n
2 2
e.g.,
x = 32 + 2 2 + 5 2 = 38
!
!
1
10. Matrix Product
Product:
A and B must have
compatible dimensions
In Matlab: >> A*B
Examples:
Matrix Multiplication is not commutative:
10
11. Matrix Sum
Sum:
A and B must have the
same dimensions
Example:
Determinant of a Matrix
Determinant: A must be square
Example:
11
12. Determinant in Matlab
Inverse of a Matrix
If A is a square matrix, the inverse of A, called A-1,
satisfies
AA-1 = I and A-1A = I,
Where I, the identity matrix, is a diagonal matrix
with all 1’s on the diagonal.
"1 0 0%
"1 0% $ '
I2 = $ ' I3 = $0 1 0'
#0 1&
$0 0 1'
# &
!
!
12
13. Inverse of a 2D Matrix
Example:
Inverses in Matlab
13
17. Some Properties of
Eigenvalues and Eigenvectors
– If λ1, …, λn are distinct eigenvalues of a matrix, then
the corresponding eigenvectors e1, …, en are linearly
independent.
– A real, symmetric square matrix has real eigenvalues,
with eigenvectors that can be chosen to be orthonormal.
Linear Independence
• A set of vectors is linearly dependent if one of
the vectors can be expressed as a linear
combination of the other vectors.
Example: "1% "0% "2%
$' $' $'
$0' , $1' , $1'
$' $' $'
#0& #0& #0&
• A set of vectors is linearly independent if none
of the vectors can be expressed as a linear
!
combination of the other vectors.
Example: "1% "0% "2%
$' $' $'
$0' , $1' , $1'
$' $' $'
#0& #0& #3&
!
17
18. Rank of a matrix
• The rank of a matrix is the number of linearly
independent columns of the matrix.
Examples: "1 0 2%
$
0 1 1
' has rank 2
$ '
$0 0 0'
# &
"1 0 2%
$ '
! $0 1 0' has rank 3
$0 0 1'
# &
• Note: the rank of a matrix is also the number of
linearly!independent rows of the matrix.
Singular Matrix
All of the following conditions are equivalent. We
say a square (n × n) matrix is singular if any one
of these conditions (and hence all of them) is
satisfied.
– The columns are linearly dependent
– The rows are linearly dependent
– The determinant = 0
– The matrix is not invertible
– The matrix is not full rank (i.e., rank < n)
18
19. Linear Spaces
A linear space is the set of all vectors that can be
expressed as a linear combination of a set of basis
vectors. We say this space is the span of the basis
vectors.
– Example: R3, 3-dimensional Euclidean space, is
spanned by each of the following two bases:
"1% "0% "0% "1% "0% "0%
$' $' $' $' $' $'
$0' , $1' , $0' $0' , $1' , $0'
$' $' $'
#0& #0& #1& $' $' $'
#0& #2& #1&
! !
Linear Subspaces
A linear subspace is the space spanned by a subset
of the vectors in a linear space.
– The space spanned by the following vectors is a
two-dimensional subspace of R3.
"1% "0%
$' $'
What does it look like?
$0' , $1'
$0' $0'
#& #&
– The space spanned by the following vectors is a
two-dimensional subspace of R3.
! "1% "0%
$' $' What does it look like?
$1' , $0'
$' $'
#0& #1&
!
19
20. Orthogonal and Orthonormal
Bases
n linearly independent real vectors
span Rn, n-dimensional Euclidean space
• They form a basis for the space.
– An orthogonal basis, a1, …, an satisfies
ai ⋅ aj = 0 if i j
– An orthonormal basis, a1, …, an satisfies
ai ⋅ aj = 0 if i j
ai ⋅ aj = 1 if i = j
– Examples.
Orthonormal Matrices
A square matrix is orthonormal (also called
unitary) if its columns are orthonormal vectors.
– A matrix A is orthonormal iff AAT = I.
• If A is orthonormal, A-1 = AT
AAT = ATA = I.
– A rotation matrix is an orthonormal matrix with
determinant = 1.
• It is also possible for an orthonormal matrix to have
determinant = -1. This is a rotation plus a flip (reflection).
20
21. SVD: Singular Value Decomposition
Any matrix A (m × n) can be written as the product of three
matrices:
A = UDV T
where
– U is an m × m orthonormal matrix
– D is an m × n diagonal matrix. Its diagonal elements, σ1, σ2, …, are
called the singular values of A, and satisfy σ1 ≥ σ2 ≥ … ≥ 0.
– V is an n × n orthonormal matrix
Example: if m > n
A U D VT
"• • •% "( ( ( (% "*1 0 0 %
$ ' $ ' $ '"
$• • •' $ | | | |' $ 0 * 2 0 '$+ v1
T
,%
'
$• • •' = $u1 u2 u3 L um ' $ 0 0 * n '$ M M M'
$ ' $ ' $ '$
$• • •' $ | | | |' $ 0 0 0 '#+ v n
T
,'
&
$
#• ' $
• •& #) ) ) )& ' $
#0 0 0 ' &
!
SVD in Matlab
>> x = [1 2 3; 2 7 4; -3 0 6; 2 4 9; 5 -8 0]
x=
1 2 3
2 7 4
-3 0 6
2 4 9
5 -8 0
>> [u,s,v] = svd(x)
u= s=
-0.24538 0.11781 -0.11291 -0.47421 -0.82963 14.412 0 0
-0.53253 -0.11684 -0.52806 -0.45036 0.4702 0 8.8258 0
-0.30668 0.24939 0.79767 -0.38766 0.23915 0 0 5.6928
-0.64223 0.44212 -0.057905 0.61667 -0.091874
0 0 0
0.38691 0.84546 -0.26226 -0.20428 0.15809
0 0 0
v=
0.01802 0.48126 -0.87639
-0.68573 -0.63195 -0.36112
-0.72763 0.60748 0.31863
21
22. Some Properties of SVD
– The rank of matrix A is equal to the number of nonzero
singular values σi
– A square (n × n) matrix A is singular iff at least one of
its singular values σ1, …, σn is zero.
Geometric Interpretation of SVD
If A is a square (n × n) matrix,
A U D VT
"• • •% " ( L ( % "*1 0 0 %"+ v1 T
,%
$ ' $ ' $ '$ '
$• • •' = $u1 L un ' $ 0 * 2 0 '$ M M M'
$• • •'
# & $ ) L ) ' $ 0 0 * n '$+ v T
# & # &# n ,'
&
– U is a unitary matrix: rotation (possibly plus flip)
– D is a scale matrix
T
! – V (and thus V ) is a unitary matrix
Punchline: An arbitrary n-D linear transformation is
equivalent to a rotation (plus perhaps a flip), followed by a
scale transformation, followed by a rotation
Advanced: y = Ax = UDV Tx
T
– V expresses x in terms of the basis V.
– D rescales each coordinate (each dimension)
– The new coordinates are the coordinates of y in terms of the basis U
22