This document discusses using least squares approximation to fit linear and polynomial models to data. It introduces the concept of best approximation in a subspace and shows that the orthogonal projection of a vector onto the subspace is the best approximation. The document then applies these concepts to derive the normal equations and the least squares solution for linear and polynomial regression models. Examples are provided to illustrate fitting lines and curves to data using least squares.

1. Christopher Carbone
We will connect the Best Approximation Theory to Least Squares
approximation through the use of a motivating example. We will further
explore data modeling by using least squares to approximate a line of
best fit, or a curve of best fit.

2. You complete an experiment to weigh Fiddler Crabs that
were submerged in a saline solution.
Time
(minutes)
Mean Change of Weight
(grams)
0 0.0
20 0.9
40 1.1
60 1.1
80 1.1
100 1.2
Table 1

3. We want to predict the weight of the Fiddler Crabs at 120
minutes.
Take x1 to be the initial weight of the Fiddler Crabs and
take x2 to be the weight gain per minute:
x1 + 0x2 = 0.0
x1 + 20x2 = 0.9
x1 + 40x2 = 1.1
x1 + 60x2 = 1.1
x1 + 80x2 = 1.1
x1 + 100x2 = 1.2

4. To approximate the vector u on the plane W:
We will show that projWu is the best approximation of u by
vectors in W. To do this, choose a vector v∈W such that v ≠
projWu.
Approximating ||u-projWu||, amounts to minimizing ||u-v||.

5. Given a finite-dimension subspace W, we need to write a
vector u as a linear combination of vectors in the
subspace W. But, we cannot do this exactly since u∉W.
We will show that the best we can do is projWu since
for all v∈W where v ≠ projWu, ||u-projwu|| < ||u – v||.
Therefore, projwu is the “best approximation” of u by
vectors in W.

6. Proof:
 Let W be a finite-dimension subspace of an inner
product space V with u∈V and v∈W.
 projWu,v∈W, so (projWu – v)∈W.
 u – projWu is orthogonal to W, so u – projWu is
orthogonal to projWu – v as well.
 u – v = (u – projWu) + (projWu – v).
 We can apply the Pythagorean Theorem,
||u-v||2 = ||u – projWu||2 + ||projWu - v||2.
 As long as v ≠ projWu, ||u-v||2 > ||u – projWu||2.
Therefore, ||u-v|| > || u – projWu||. ☐

7. If we have a matrix A, then we can let the column space
of A be W. Thus, W represents the vectors Ax for all
x∈Rn. Since the system Ax = b is inconsistent, b∉W.
Thus, ∄x∈Rn such that Ax = b. We will find x’∈Rn such
that Ax’∈W where Ax’ is the best approximation for
b∉W.

8.  Given the inconsistent linear system Ax = b, we want
to find the least squares solution x’ to minimize
||Ax - b||. If ||Ax’ – b|| is large, then x’ is regarded as a
poor solution. If ||Ax’ – b|| is small, x’ is a good
solution. Thus, this solution x’ is called a least squares
solution of Ax = b.

9.  Let e = Ax – b. Then writing e component-wise yields
e = (e1, e2, … , em). Since we try to minimize this
vector, we are minimizing ||e|| = .
Therefore, the solution will minimize ||e||2 = (e1
2 + e2
2
+ … + em
2) as well. Thus, we are trying to find the sum
of the least squares to minimize that vector.

10.  From the Best Approximation Theorem, the closest
vector x’ is the orthogonal projection of b on W.
 For x’ to be a least squares solution to the system
Ax=b, Ax’ = projWb.
 Since b∉W, b – Ax’ = b – projWb is orthogonal to W.
 W is the column space of A, so b – Ax’ must lie in the
nullspace of AT.
 A least squares solution x’ of Ax’ = b would satisfy
AT(b – Ax’) = 0, or equivalently ATAx’ = ATb.
 The system must be consistent as we are assuming
Ax’ = b, since x’ is the least squares solution to Ax = b.

11. For an n x m matrix A, A has linearly independent
column vectors if and only if ATA is invertible.
The square matrix that is obtained from ATA is invertible
when the matrix A has linearly independent column
vectors.
Thus, from ATAx’ = ATb. We obtain x’ = (ATA)-1ATb.

12. Recalling our earlier example about the Fiddler Crabs,
the water regulation is called osmoregulation, such
that the crabs will undergo osmosis in the solution,
and thus gain weight. Let’s see the linear equations
produced again:
x1 + 0x2 = 0.0
x1 + 20x2 = 0.9
x1 + 40x2 = 1.1
x1 + 60x2 = 1.1
x1 + 80x2 = 1.1
x1 + 100x2 = 1.2

14. Given a set of data of the form (x1,y1), (x2,y2), …, (xn,yn), we
model the data by trying to fit an equation to the data.
n + 1 sets of data can be modeled by polynomials all the way
up to the nth-degree polynomial. But, when we try to fit a
curve to data, some measurement error and rounding
issues exists. Therefore, the curve cannot fit the data
exactly. So, we will use the least squares technique.

15. We minimize the error between the data points and the
actual line formed from the line of best fit. This is the
vertical distance between the points and the line,
denoted as the distance dj.
We believe that there is an additive error with the
vertical coordinates, not the horizontal coordinates,
producing yj = a0 + a1xj + dj.

16. The equation of the line would be y = a0 + a1x. So, the
resulting linear system would look like:
y1 = a0 + a1x1
y2 = a0 + a1x2
…
yn = a0 + a1xn
Or equivalently,

17. In the Example, we have carried out the least squares
solution for a straight line of best fit.
The equation would be y = 0.42857 + 0.00943x.

19. The equation is y = a0 + a1x + a2x2.
The equation is y = 0.14286 + 0.03086x – 0.00002x2.

21. The equation is y = a0 + a1x + a2x2 + a3x3.
The equation is
y = 0.0150794 + 0.0600331x – 0.0010129x2 + 0.0000053x3

23. The number of columns increased in matrix A for each
degree of the proposed polynomial function. We can
fit a polynomial of degree n to m data points as:
y1 = a0 + a1x1 + a2x1
2 + … + anx1
n
y2 = a0 + a1x2 + a2x2
2 + … + anx2
n
…
ym = a0 + a1xm + a2xm
2 + … + anxm
n
We can solve for the least squares solution x’ =
(ATA)-1ATb to fit m data points with an m – 1 degree
polynomial:

24. Anton, Howard. Elementary Linear Algebra. 9th ed. John
Wiley & Sons. United States of America. 2005.
Johnson, Arlene Prof. Animal Osmoregulation. Biology
Lab. BI114L. 26 February 2010.