linear Algebra least squares

1. Least Squares Approximation: A
Linear Algebra Technique
By Larry Wong and
James Sfregola

2. Soo……You have a bunch of
Data.
Attendence Big 4 Derby League Position per capita income adjusted population
39,968 1 0 11 35000 1582564
21,852 1 0 10 36725 137471
24,409 1 0 7 36725 371847
26,770 1 0 19 43600 852013
41,917 1 1 2 43600 1351187
37,162 1 0 6 36725 623160
24,510 1 1 16 43600 724121
43,958 1 1 3 36725 738873
40,699 1 0 14 36725 630043
75,595 1 1 1 36725 1194415
 What can we do with this?
 Use Least Squares Approximation (LSE) to estimate the
relationship of the dependent variable and the independent
variable(s).

3. What is Least Squares?
• Approximates solutions for inconsistent over
determined systems
– What does inconsistent mean?
– Over determined, what’s that?
• In other words LSE allows you to describe a
model as being represented by the line of best
fit.
• Where the “best fit” line, curve or polynomial is created from
the approximations.

4. Least Squares it’s Actually Pretty Useful
 Least Squares Approximations are
highly relevant in various fields
 Really, how so?
 Still don’t believe us?
 Fine we’ll give examples…but not now.

5. So How Does it Estimate a Solution?
 Least Squares Approximation estimates
a best fit solution for a system by
 Minimizing the magnitude error vector, e.
 Estimating the coefficients of the best fit
equation

6. How Do We Ensure Positive Error?
 Least Squares
– Eliminates negative terms
– More susceptible to outlying data
 Least Absolute Difference
– Difficult to work with in linear algebra

7. `Petal L.`
`Petal
W.`
1 2 3 4 5 6 7
0.5
1.0
1.5
2.0
2.5

8. The Least Squares Theorem
 For an over determined inconsistent
system Ax=b:
 This is easily solved using inverse and
transpose multiplication
 the resulting vector x can be determined in
the form x=(AT A) -1 AT b
 where x will correspond to the coefficients in
your linear or polynomial expression.

9. How to Approximate the Error Term?
• e = b-Ax , where e is
the error term
• Where e1= the
distance from our
data point to the
best fit
approximation
given by the
orthogonal
projection

10. Application to Economics
 Econometrics utilizes Ordinary Least
Squares Approximations (OLS)
 Similar to LSE except it follows 7 classical
assumptions
Estimated Regression Equation

11. -30000
-20000
-10000
0
10000
20000
30000
0
20000
40000
60000
80000
50 100 150 200 250 300 350
Residual Actual Fitted

13. Bibliography
• Poole, David Linear Algebra: A Modern
Introduction. Canada: Thomson
Brooks/Cole, 2006.
• Studenmund, A.H. Using Econometrics: A
Practical Guide. New York: Pearson
Education, 2006.