This document provides an overview of multiple regression analysis. It discusses estimating multiple regression models, interpreting estimated coefficients, omitted variable bias, goodness of fit measures like R-squared, assumptions of the model including exogeneity of regressors and homoskedasticity, variance of OLS estimators, and the Gauss-Markov theorem establishing OLS as the best linear unbiased estimator under the assumptions.

1. Multiple Regression Analysis
Võ Đức Hoàng Vũ
University of Economics HCMC
June 2015
Võ Đức Hoàng Vũ (UEH) Applied Econometrics June 2015 1 / 1

2. Estimation
The model is : y = β0 + β1x1 + β2x2 + . . . + βkxk
β0 is still the intercept
β1 to βk all called slope parameters
u is still the error term (or disturbance)
Still need to make a zero conditional mean assumption, so now
assume that E(u|x1, x2, . . . , xk) = 0
Still minimizing the sum of squared residuals, so have k + 1 first
order conditions
Võ Đức Hoàng Vũ (UEH) Applied Econometrics June 2015 2 / 1

3. Interpreting Multiple Regression
ˆβ0 + ˆβ1x1 + . . . + ˆβkxk, so
∆ˆy = ∆ˆβ1x1 + ∆ˆβ2x2 + . . . + ∆ˆβkxk,
so holding x2, . . . , xk fixed implies that ∆ˆy = ∆ˆβ1x1, that is
each β has a ceteris paribus interpretation.
Võ Đức Hoàng Vũ (UEH) Applied Econometrics June 2015 3 / 1

4. "Partialling Out" Interpretation
Consider the case where k = 2, i.e.
ˆy = ˆβ0 + ˆβ1x1 + ˆβ2x2, then
ˆβ1 = ( ˆri1yi )/ ˆr2
i1, where ˆri1 are the residuals from the
estimated regression ˆx1 = ˆγ0 + ˆγ2x2
Previous equation implies that regressing y on x1 and x2 gives
same effect of x1 as regressing y on residuals from a regression
of x1 on x2.
This means only the part of xi1 that is uncorrelated with xi2 are
being related to yi so we’re estimating the effect of x1 on y after
x2 has been "partialled out".
Võ Đức Hoàng Vũ (UEH) Applied Econometrics June 2015 4 / 1

5. Simple vs Multiple Regression Estimate
Compare the simple regression ˜y = ˜β0 + ˜β1x1 with the multiple
regression haty = ˆβ0 + ˆβ1x1 + ˆβ2x2
Generally, ˜β1 = ˆβ1 unless: ˆβ2 = 0 (i.e. no partial effect of x2) OR
x1 and x2 are uncorrelated in the sample
Võ Đức Hoàng Vũ (UEH) Applied Econometrics June 2015 5 / 1

6. Goodness-of-fit
We can think of each observation as being made up of an
explained part, and an unexplained part - yi = ˆyi + ˆui - We then
define the following:
(yi − ¯y)2
is the total sum of squares (SST)
(ˆyi − ¯y)2
is the explained sum of squares (SSE)
ˆu2
i is the residual sum of squares (SSR)
Then SST = SSE + SSR
How do we think about how well our sample regression line fits
our sample data?
Can compute the fraction of the total sum of squares (SST) that
is explained by the model, call this the R-squared of regression
R2
= SSE/SST = 1 − SSR/SST
Võ Đức Hoàng Vũ (UEH) Applied Econometrics June 2015 6 / 1

7. Goodness-of-fit (cont)
We can also think of R2
as being equal to the squared
correlation coefficient between the actual yi and the value ˆyi
R2
=
( (yi − ¯y)(ˆyi − ¯ˆy))2
( (yi − ¯y)2)( (ˆyi − ¯ˆy)2)
R2
can never decrease when another independent variable is
added to regression, and usually will increase
Because R2
will usually increase with the number of independent
variables, it is not a good way to compare models.
Võ Đức Hoàng Vũ (UEH) Applied Econometrics June 2015 7 / 1

8. Assumption for Unbiasedness
Population model is linear in parameters:
y = β0 + β1x1 + β2x2 + . . . + βkxk + u
We can use a random sample of size
n, {(xi1, xi2, . . . , xik, yi ) : i = 1, 2, . . . , n}, from the population
model, so that the sample model is
yi = β0 + β1xi1 + β2xi2 + . . . + βkxik + ui
E(u|x1, x2, . . . , xk) = 0, implying that all of the explanatory
variables are exogenous.
None of the x’s is constant, and there are no exact linear
relationships among them.
Võ Đức Hoàng Vũ (UEH) Applied Econometrics June 2015 8 / 1

9. Too Many or Too Few Variables
What happens if we include variables in our specification that
don’t belong?
There is no effect on our parameter estimate, and OLS remains
unbiased
What if we exclude a variable from our specification that does
belong?
OLS will usually be biased.
Võ Đức Hoàng Vũ (UEH) Applied Econometrics June 2015 9 / 1

10. Omitted Variable Bias
Suppose the true model is given as
y = β0 + β1x1 + β2x2 + u, but we estimate ˜y = + ˜β1x1 + u, then
˜β1 =
(xi1 − ¯x1)yi
(xi1 − ¯x1)2
Recall the true model, so that yi = β0 + β1xi1 + β2xi2 + ui , so the
numerator becomes
(xi1 − ¯x1)(β0 + β1xi1 + β2xi2 + ui ) =
β1 (xi1 − ¯x1)2
+ β2 (xi1 − ¯x1)xi2 + (xi1 − ¯x)ui
Võ Đức Hoàng Vũ (UEH) Applied Econometrics June 2015 10 / 1

11. Omitted Variable Bias (cont)
˜β = β1 + β2
(xi1 − ¯x1)xi2
(xi1 − ¯x1)2
+
(xi1 − ¯x1)ui
(xi1 − ¯x1)2
since E(ui ) = 0, taking expectations we have
E( ˜β1) = β1 + β2
(xi1 − ¯x1)xi2
(xi1 − ¯x1)2
Consider the regression of x2 on x1
˜x2 = ˜δ0 + ˜δ1x1 then ˜δ1 =
(xi1 − ¯x1)xi2
(xi1 − ¯x1)2
so E(˜β1) = β1 + β2
˜δ1
Võ Đức Hoàng Vũ (UEH) Applied Econometrics June 2015 11 / 1

12. Summary of Direction of Bias
Corr(x1, x2) > 0 Corr(x1, x2) < 0
β2 > 0 Positive bias Negative bias
β2 < 0 Negative bias Positive bias
Võ Đức Hoàng Vũ (UEH) Applied Econometrics June 2015 12 / 1

13. Omitted Variable Bias Summary
Two cases where bias is equal to zero
β2 = 0 that is x2 doesn’t really bebong in model
x1 and x2 are uncorrelated in the sample
If correlation between x2, x1 and x2, y is the same direction, bias
will be positive
If correlation between x2, x1 and x2, y is the opposite direction,
bias will be negative
Technically, can only sign the bias for the more general case if al
of the included x’s are uncorrelated.
Typpically, then, we work through the bias assuming the x’s are
uncorrelated, as a useful guide even if this assumption is not
strictly true.
Võ Đức Hoàng Vũ (UEH) Applied Econometrics June 2015 13 / 1

14. Variance of the OLS Estimators
Now we know that the sampling distribution of our estimate is
centered around the true parameter
Want to think about how spread out this distribution is
Much easier to think about this variance under an additional
assumption, so
Assume Var(u|x1, x2, . . . , xk) = σ2
(Homoskedasticity)
Let xxx stand for (x1, x2, . . . , xk)
Assuming that Var(u|x) = σ2
also implies that Var(y|x) = σ2
The 4 assumptions for unbiasedness, plus this homoskedasticity
assumption are known as the Gauss-Markov assumptions
Võ Đức Hoàng Vũ (UEH) Applied Econometrics June 2015 14 / 1

15. Variance of OLS (cont.)
Given the Gauss-Markov assumptions
Var( ˆβj) =
σ2
SSTj(1 − R2
j )
, where
SSTj = (xij − ¯xj)2
and R2
j is the R2
from
regressioing xj on all other x’s
Võ Đức Hoàng Vũ (UEH) Applied Econometrics June 2015 15 / 1

16. Components of OLS Variances
The error variance: a larger σ2
implies a larger variance for the
OLS estimators
The total sample variation: a larger SSTj implies a smaller
variance for the estimators
Linear relationships among the independent variables: a larger
R2
j implies a larger variance for the estimators
Võ Đức Hoàng Vũ (UEH) Applied Econometrics June 2015 16 / 1

17. Misspecified Models
Consider again the misspecified model
˜y = ˜β0 + ˜β1x1, so that Var( ˜β1) =
σ2
SST1
Thus, Var( ˜β1) < Var( ˆβ1) unless x1 and x2 are
uncorrelated, then they’re the same
Võ Đức Hoàng Vũ (UEH) Applied Econometrics June 2015 17 / 1

18. Misspecified Models (cont.)
While the variance of the estimator is smaller for the misspecified
model, unless β2 = 0 the misspecified model is biased
As the sample size grows, the variance of each estimator shrinks
to zero, making the variance difference less important
Võ Đức Hoàng Vũ (UEH) Applied Econometrics June 2015 18 / 1

19. Estimating the Error Variance
We don’t know what the error variance, σ2
, is, because we don’t
observe the errors, ui
What we observe are the residuals, ˆui
We can use the residuals to form an estimate of the error
variance
ˆσ2
= ( ˆu2
i )/(n − k − 1) ≡ SSR/df
thus, se(ˆβj ) = ˆσ/[SSTj (1 − R2
j )]1/2
df = n − (k + 1)
Võ Đức Hoàng Vũ (UEH) Applied Econometrics June 2015 19 / 1

20. The Gauss-Markov Theorem
Given our 5 Gauss-Markov Assumption it can be shown that
OLS is "BLUE"
Best
Linear
Unbiased
Estimator
Thus, if the assumptions hold, use OLS
Võ Đức Hoàng Vũ (UEH) Applied Econometrics June 2015 20 / 1