2. ε
X
β
β
Y 1
0
Linear component
Population Regression model
Population
y intercept
Population
Slope
Coefficient
Random
Error
term, or
residual
Dependent
Variable
Independent
Variable
Random Error
component
2
3. Linear Regression Assumptions
• Error values (ε) are statistically independent
• Error values are normally distributed for any given value of x
• The probability distribution of the errors is normal
• The probability distribution of the errors has constant variance
• The underlying relationship between the x variable and the y
variable is linear
3
5. x
b
b
ŷ 1
0
i
The sample regression line provides an estimate of
the population regression line
Estimated Regression Model
Estimate of
the regression
intercept
Estimate of the
regression slope
Estimated
(or predicted)
y value
Independent
variable
The individual random error terms ei have a mean of zero
5
6. Scatter plot
• Plot of All (Xi, Yi) Pairs
• Suggests How Well Model Will Fit
0
20
40
60
0 20 40 60
X
Y
6
7. Thinking Challenge
• How would you draw a line through the points? How do you determine
which line ‘fits best’?
0
20
40
60
0 20 40 60
X
Y
7
8. Thinking Challenge
• How would you draw a line through the points? How do you determine
which line ‘fits best’?
0
20
40
60
0 20 40 60
X
Y
Slope changed
Intercept unchanged
8
9. Thinking Challenge
• How would you draw a line through the points? How do you determine
which line ‘fits best’?
Slope changed
0
20
40
60
0 20 40 60
X
Y
Intercept changed
9
10. Least Squares
• 1. ‘Best Fit’ Means Difference Between Actual Y Values & Predicted Y
Values Are a Minimum. So square errors!
• 2. LS Minimizes the Sum of the Squared Differences (errors) (SSE)
n
i
i
n
i
i
i Y
Y
1
2
1
2
ˆ
ˆ
10
12. Example
A study was made by a retail merchant to determine the relation between
weekly advertising expenditures (X) and sales (Y). Estimate regression line to
predict weekly sales from advertising expenditures and interpret it. Predict the
Sales for a weekly expenditures 29. Also test the Significance of the model and
find R2 and interpret it.
12
Y X
385 20
400 15
395 25
440 40
𝑛 = 4
𝑌 =
𝑌
𝑛
= 405
𝑋 =
𝑋
𝑛
= 25
1620 100
13. Example - Computations
• Computation of Simple Linear Regression equation
13
Y X
385 20
400 15
395 25
440 40
1620 100
14. Example - Computations
• Computation of Simple Linear Regression equation
14
Y X x
385 20 -5
400 15 -10
395 25 0
440 40 15
1620 100 0
15. Example - Computations
• Computation of Simple Linear Regression equation
15
Y X x y
385 20 -5 -20
400 15 -10 -5
395 25 0 -10
440 40 15 35
1620 100 0 0
16. Example - Computations
• Computation of Simple Linear Regression equation
16
Y X x y xy
385 20 -5 -20 100
400 15 -10 -5 50
395 25 0 -10 0
440 40 15 35 525
1620 100 0 0 675
17. Example - Computations
• Computation of Simple Linear Regression equation
17
Y X x y xy x2
385 20 -5 -20 100 25
400 15 -10 -5 50 100
395 25 0 -10 0 0
440 40 15 35 525 225
1620 100 0 0 675 350
18. Example - Computations
• Computation of Simple Linear Regression equation
18
Y X x y xy x2 y2
385 20 -5 -20 100 25 400
400 15 -10 -5 50 100 25
395 25 0 -10 0 0 100
440 40 15 35 525 225 1225
1620 100 0 0 675 350 1750
20. Interpretation
20
•The value of b1=1.93, indicates that the average sales are
expected to increase by 1.93 Rs. With each one Rs. increase
in the Advertising Expenditures.
•The value of b0=356.75 indicates the average sales without
any expenditures on Advertisements. The interpretation of
b0 is not always meaningful.
𝑺𝒂𝒍𝒆𝒔 = 356.75 + 1.93 𝐀𝐝𝐯. 𝐄𝐱𝐩𝐞𝐧𝐝𝐢𝐭𝐮𝐫𝐞𝐬
𝒀 = 356.75 + 1.93 𝐗
23. Test of Hypothesis for 𝛽1
23
𝑡 =
𝒃𝟏 − 𝜷𝟏
𝑺𝑬(𝒃𝟏)
=
𝟏. 𝟗𝟑 − 𝟎
𝟐𝟐𝟒. 𝟏𝟏
𝟑𝟓𝟎
= 𝟐. 𝟒𝟏
Table value
𝑡𝛼
2
,(𝑛−2)
= 4.303
Test statistic
Conclusion: We have not sufficient evidence from the sample to reject
the Null Hypothesis since the calculated value is not greater than the
table value.
Interpretation: Therefore, the Advertising expenditures do not have
significant effect on Sales at 5% level of significance.
25. Confidence Interval for 𝜷𝟏
• b1 is the estimate of 𝜷𝟏
• SE(b1), already computed
• t is the table value
• There will be two limits of the confidence interval,
the lower limit and the upper limit
25
𝑏1 ± 𝑡𝛼
2
,(𝑛−2)𝑆𝐸(𝑏1)
28. Confidence Interval for 𝜷𝟎
• b0 is the estimate of 𝜷𝟎
• SE(b0), already computed
• t is the table value
• There will be two limits of the confidence interval,
the lower limit and the upper limit
28
𝑏0 ± 𝑡𝛼
2
,(𝑛−2)𝑆𝐸(𝑏0)
29. ANOVA
• Partition of total variation in Response Variable Y into two components i.e.,
explained (due to Regression) and the unexplained (Residual) variation. Explained
variation is the variation due to regression i.e., variation due to X and the
unexplained variation is the variation due to uncontrolled factors other than X.
TSS = Regression SS + Residual SS
𝑇𝑆𝑆 = 𝑦2
𝑅𝑒𝑔𝑟𝑒𝑠𝑠𝑖𝑜𝑛 𝑆𝑆 = 𝑏1 × 𝑥 𝑦
𝑅𝑒𝑠𝑖𝑑𝑢𝑎𝑙 𝑆𝑆 = 𝑇𝑆𝑆 − 𝑅𝑒𝑔𝑟𝑒𝑠𝑠𝑖𝑜𝑛 𝑆𝑆
29
30. Calculations 𝑇𝑆𝑆 = 𝑦2 = 1750
𝑅𝑒𝑔𝑟𝑒𝑠𝑠𝑖𝑜𝑛 𝑆𝑆 = 𝑏1 × 𝑥 𝑦 = 1301.79
𝑅𝑒𝑠𝑖𝑑𝑢𝑎𝑙 = 𝑇𝑆𝑆 − 𝑅𝑒𝑔𝑟𝑒𝑠𝑠𝑖𝑜𝑛 𝑆𝑆 = 448.21
We will now construct ANOVA table to test the hypothesis that 𝛽1 = 0.
Source of
variation
(S.O.V)
Degree of
freedom (df)
Sum of squares
(SS)
Mean sum of
squares
MSS=SS/df
Fcal Ftab
Regression 1 1301.79 1301.79 5.80 18 . 513
Residual 2 448.21 224.11
Total 3 1750
As the calculated value of F is not greater than the table value, therefore, we
have not sufficient evidence against the Null Hypothesis and conclude that
the Sales have not been affected significantly from Adv. Expenditures. 30
31. Goodness of Fit (R2)
• A commonly used measure of the goodness of fit of a linear model is R2 called
coefficient of determination.
• The co-efficient of determination tells us the proportion of variation in the
response variable explained by the independent variable.
𝑅2
=
𝐸𝑥𝑝𝑙𝑎𝑖𝑛𝑒𝑑 𝑉𝑎𝑟𝑖𝑎𝑡𝑖𝑜𝑛
𝑇𝑜𝑡𝑎𝑙 𝑉𝑎𝑟𝑖𝑎𝑡𝑖𝑜𝑛
× 100
𝑹𝟐
=
1301.79
1750
× 100 = 74.39%
• The Advertisement Expenditures (X) explains 74.39% of the variation in Sales (Y)
and the rest is due to some unknown factors.
31
32. Example
Following data shows the Revenue of six firms in (000)$ along with the
expenditures on Research & Development (000)$.
• Draw a scatter plot and assess the relationship between Y and X.
• Fit simple linear regression equation and interpret the parameters.
• Test the hypothesis that there is no linear relation between Y and X i.e., β1=0. Also
compute 95% Confidence Interval for β1.
• Test the hypothesis that β0 > 15. Compute 95% Confidence Interval for β0.
• Perform analysis of variance (ANOVA) and test the significance of the regression
model. Calculate coefficient of determination and interpret it.
• Test the hypothesis that the mean revenue for firm at X=9 is greater than 30 i.e.,
𝜇𝑌
𝑋=9
> 30. Also construct 95% Confidence Interval.
Revenue in (000) $ Y 31 40 30 34 25 20
Expenditure on R & D in (000) $ X 5 11 4 5 3 2
32
33. Interpretation
• The value of b1=2, indicates that the average yield of rice is expected to
increase by 2 maunds with each one kg increase in the fertilizer.
• The value of b0=20 indicates that average yield of rice will be 20 kg without
using the fertilizer. The interpretation of b0 is not always meaningful.
𝒀 = 20 + 2 𝑿
Muhammad Usman 38
34. ANOVA
• Partition of total variation in Response Variable Y into two components i.e.,
explained (due to Regression) and the unexplained (Residual) variation. Explained
variation is the variation due to regression i.e., variation due to X and the
unexplained variation is the variation due to uncontrolled factors other than X.
TSS = Regression SS + Residual SS
𝑇𝑆𝑆 = 𝑦2
𝑅𝑒𝑔𝑟𝑒𝑠𝑠𝑖𝑜𝑛 𝑆𝑆 = 𝑏1 × 𝑥 𝑦
𝑅𝑒𝑠𝑖𝑑𝑢𝑎𝑙 𝑆𝑆 = 𝑇𝑆𝑆 − 𝑅𝑒𝑔𝑟𝑒𝑠𝑠𝑖𝑜𝑛 𝑆𝑆
46
35. Calculations 𝑇𝑆𝑆 = 𝑦2 = 242
𝑅𝑒𝑔𝑟𝑒𝑠𝑠𝑖𝑜𝑛 𝑆𝑆 = 𝑏1 × 𝑥 𝑦 = 200
𝑅𝑒𝑠𝑖𝑑𝑢𝑎𝑙 𝑆𝑆 = 𝑇𝑆𝑆 − 𝑅𝑒𝑔𝑟𝑒𝑠𝑠𝑖𝑜𝑛 𝑆𝑆 = 242 − 200 = 42
We will now construct ANOVA table to test the hypothesis that 𝛽1 = 0.
Source of
variation (S.O.V)
Degree of
freedom (df)
Sum of squares
(SS)
Mean sum of
squares
MSS=SS/df
Fcal Ftab
Regression 1 200 200.00 19.05 7.709
Error 4 42 10.50
Total 5 242
As the calculated value of F is greater than the table value of F i.e., 19.05 > 7.709,
therefore, we will reject the Null Hypothesis that the 𝛽1=0 and conclude that the
relationship between Y and X is significant. 47
36. Goodness of Fit (R2)
• A commonly used measure of the goodness of fit of a linear model is R2 called
coefficient of determination.
• The co-efficient of determination tells us the proportion of variation in the
dependent variable explained by the independent variable.
𝑅2
=
𝐸𝑥𝑝𝑙𝑎𝑖𝑛𝑒𝑑 𝑉𝑎𝑟𝑖𝑎𝑡𝑖𝑜𝑛
𝑇𝑜𝑡𝑎𝑙 𝑉𝑎𝑟𝑖𝑎𝑡𝑖𝑜𝑛
× 100
𝑹𝟐
=
200
242
× 100 = 82.64
• The R& D expenditures (X) explains 82.64% of the variation in Revenue (Y) and the
rest is due to some unknown factors.
48
37. Test of hypothesis for
49
𝐻0: 𝜇𝑦/𝑥=9 ≤ 30 𝑣𝑠 𝐻1: 𝜇𝑦/𝑥=9 > 30
𝑡 =
𝑌𝑥 − 𝜇𝑦 𝑥
𝑆𝐸 𝑌𝑥
, 𝑤ℎ𝑒𝑟𝑒 SE 𝑌𝑥 = 𝑺𝟐
𝒀.𝑿
1
𝑛
+
𝑋0 − 𝑋 2
𝒙𝟐
