Simple linear Regression

1. SIMPLE LINEAR REGRESSION
PREPARED BY
HASSAN SHEHWAR SHAH
DEPARTMENT OF MECHANICAL ENGINEERING
UNIVERSITY OF ENGINEERING AND TECHNOLOGY
TAXILA
3/16/2021 1

2. OUTLINES
 Introduction of linear regression
 Simple linear regression Definition
 Simple linear Regression Model
 Assumption of Simple linear Regression
 Formulas for Estimate Regression Line
 Example
3/16/2021 2
Department of Mechanical Engineering

3. INTRODUCTION TO LINEAR REGRESSION
 Linear Regression Is Used To:
‒ Predict The Value Of A Dependent Variable Based On The Value Of At Least One
Independent Variable
‒ Explain The Impact Of Change In An Independent Variable On The Dependent
Variable
Dependent Variable : The variable we wish to predict or explain
Independent Variable : The variable used to explain the dependent variable
3/16/2021 3
Department of Mechanical Engineering

4. SIMPLE LINEAR REGRESSION MODEL
 Only One Independent Variable, X
 Relationship Between X And Y Is Described By A Linear Function
 Changes in Y Are Assumed To Be Caused By Changes in X
3/16/2021 4
Department of Mechanical Engineering

5. SIMPLE LINEAR REGRESSION MODEL
Yi = β0 + β1Xi + Ԑi
 This is also known as Population Regression Model (PRM)
Where Yi Dependent Variable or observation randomly drawn from Population
Xi Independent variable or fixed
Ԑi Random error or Residual
β0 & β1 Population Parameters
β0 is Intercept β1 Slope or Regression Coefficient
3/16/2021 5
Department of Mechanical Engineering

6. SIMPLE LINEAR REGRESSION MODEL
3/16/2021 6
Y
Observed Value
of Y for Xi
εi
Slope = β1
Predicted Value
of Y for Xi
Random Error
for this Xi value
Intercept = β0
Xi
Yi = β0 + β1Xi
X
Department of Mechanical Engineering

7. TYPES OF RELATIONSHIPS
Y
X
Y
X
Y
Y
X
X
Strong relationships Weak relationships
3/16/2021 7
Department of Mechanical Engineering

8. ASSUMPTION OF SIMPLE LINEAR REGRESSION
3/16/2021 8
• The simple linear Regression model is linear parameters.
• The Independent variable X values are non random.
• The residual (error) term has zero mean or the expected value of error term is zero.
• The residual (error) values follow the normal distribution
• There is no relationship between residual (error) term and X variable
Department of Mechanical Engineering

9. ESTIMATE REGRESSION MODEL
 Let there be a set of observations (Xi, Yi) , i = 1,2,3-----n,
Yi = β0 + β1Xi + Ԑi
In terms of sample data
Yi = b0 + b1Xi + ei
Where “b0” and “b1” are least square estimate of β0 and β1 and ei commonly called
Residual.
3/16/2021 9
Department of Mechanical Engineering

10. FORMULA FOR ESTIMATION REGRESSION LINE
• The least square estimates are as under
b1 =
𝑋𝑖
𝑌𝑖
−𝑛𝑋Y
𝑋𝑖
2
−𝑛𝑋2 (1)
b0 = Y̅ - b1X̅ (2)
Where X̅ =
𝑋𝑖
𝑛
& Y̅ =
𝑌𝑖
𝑛
the parameters b0 and b1 can also be expressed in terms of sums of squares as
follows:
b1 =
𝑆𝑆𝑋𝑌
𝑆𝑆𝑋
(3)
b0 = Y̅ - b1X̅ (4)
3/16/2021 10
Department of Mechanical Engineering

11. FORMULA FOR ESTIMATION REGRESSION LINE
SSx = ( 𝑋𝑖 − X̄)2 = 𝑋𝑖
2 − 𝑛X̄ (5)
SSY = (𝑌𝑖 − Ȳ)2 = 𝑌𝑖
2 − 𝑛Ȳ (6)
SSxy = (𝑋𝑖 − X̄)(Y𝑖 − Ȳ) = 𝑋𝑖𝑌𝑖 −
( Xi
)( Yi
)
n
(7)
 SSX and SSY are the terms used to determine the variance of X and variance of Y
respectively. SSX and SSY are called corrected sum of squares. And ꞵ1 and ꞵ0 are
estimated as follows:
ꞵ1 =
𝑆𝑆𝑋𝑌
𝑆𝑆𝑋
(8)
ꞵ0 = Y̅ - ꞵ1X̅ (9)
Y = ꞵ0 + ꞵ1x is the regression equation (10)
Variance of X(Sx
2) = SSx/(n-1) and (11)
Variance of Y(Sx
2) = SSY/(n-1) (12)
3/16/2021 11
Department of Mechanical Engineering

12. STANDARD ERROR AND ANOVA EQUATION
• Similarly, we can write error sum of squares (SSe) for the regression as
SSe = 𝑒𝑖
2 and Standard error as Se =
𝑒𝑖
2
𝑛−2
=
𝑆𝑆𝑒
𝑑𝑓𝑒
where, dfe are error degree of freedom
 The ANOVA equation for linear regression is
Total corrected sum of square = sum of squares due to regression + sum of square
due to error
SSTotal = SSR + SSe
 The sum of squares are computed as follows:
Let CF (Corrected factor) =
𝐺𝑟𝑎𝑛𝑑 𝑇𝑜𝑡𝑎𝑙 2
𝑁𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑂𝑏𝑒𝑟𝑣𝑎𝑡𝑖𝑜𝑛
SSTotal = 𝑌𝑖
2 −
( 𝑌𝑖
)2
𝑛
= 𝑌𝑖
2 − CF (13)
SSR = ꞵ1SSXY (14)
3/16/2021 12
Department of Mechanical Engineering

13. ANVOVA TABLE FOR SIMPLE LINEAR
REGRESSION AND F-TEST
Source Sum of Square Degree of Freedom Mean square F0
Regression SSR= ꞵ1SSXY
1 MSR=
SSR
1
𝑀𝑆𝑅
𝑀𝑆𝑒
Error SSe = SSTotal- SSR n-2 MSe =
SSe
(𝑛−2)
Total SSTotal = SSY n-1
3/16/2021 13
 Test for significance of regression is to determine if there is a linear relationship between X
and Y.
The appropriate hypotheses are H0 : ꞵ1 = 0
H1 : ꞵ1 ≠ 0
Reject H0 if F0 exceeds Fα,1,n-2
Rejection of H0 implies that there is significant relationship between the variable X and Y. we can
also test the coefficients ꞵ0 and ꞵ1 using t test with n – 2 degrees of freedom.
Department of Mechanical Engineering

14. T-TEST FOR ꞵ0 AND ꞵ1
• The hypotheses are
H0 : ꞵ0 = 0
H1 : ꞵ0 ≠ 0
And H0 : ꞵ1 = 0
H1 : ꞵ1 ≠ 0
 The test statistics for Intercept t0 =
ꞵ0
𝑀𝑆𝑒
(
1
𝑛
+
X̅2
𝑆𝑆𝑋
)
(15)
Reject H0, if |t0| > tα/2,n–2
 The test statistics for Slope t0 =
ꞵ1
𝑀𝑆𝑒/𝑆𝑆𝑋
(16)
Reject H0, if |t0| > tα/2,n–2
3/16/2021 14
Department of Mechanical Engineering

15. COEFFICIENT OF DETERMINATION (R2)
• The coefficient of determination is
R2 =
𝑆𝑆𝑅
𝑆𝑆𝑇𝑜𝑡𝑎𝑙
= 1 −
𝑆𝑆𝑒
𝑆𝑆𝑇𝑜𝑡𝑎𝑙
(17)
 The value of R2 lie between 0 and 1.
 The higher R2 the greater the percentage of the variation of explained by the
regression plain that is the better the goodness of fit of the regression plain to the
sample observations.
 The closer R2 to zero the worse the fit.
3/16/2021 15
Department of Mechanical Engineering

16. CONFIDENCE INTERVALS ON THE SLOPE AND
INTERCEPT
 A 100(1 − α)% confidence interval on the slope β1 in simple linear
regression is
β1 ± ta/2, n-2
𝑀𝑆𝑒
𝑆𝑆𝑋
(18)
 Similarly, a 100(1− α)% confidence interval on the intercept β0 is
Β0 ± ta/2, n-2 𝑀𝑆𝑒(
1
𝑛
+
X̄
𝑆𝑆𝑋
) (19)
 A100(1− α)% prediction interval on a future observation Y0 at the value X0 is
given by
Y0 ± tα/2, n-2 𝑀𝑆𝑒[1 +
1
𝑛
+
𝑋0
−X̄ 2
𝑆𝑆𝑋
] (20)
3/16/2021 16
Department of Mechanical Engineering

17. EXAMPLE SIMPLE LINEAR REGRESSION
A software company wants to find out whether their profit is related to the investment made in
their research and development. They have collected the following data from their company
records.
i. Develop a simple linear regression model to
the data and estimate the profit when the
investment is 13 Lakh rupees.
i. Test the significance of regression using F-test.
ii. Test significance of ꞵ1.
3/16/2021 17
Sr. No. Investment in R&D ( in Lakhs) Annual Profit ( in LakhS)
1 2 24
2 3 25
3 4 31
4 5 34
5 11 40
6 3 31
7 10 36
8 8.5 36
9 4 29
10 6.5 33
11 8 37
12 9.5 37
13 11.5 39
14 10.5 39
15 9.5 36
Department of Mechanical Engineering

18. SOLUTION EXAMPLE
 In this problem, the profit depends on investment on R&D. Hence, X = Investment in
R&D and Y = Profit
3/16/2021 18
X̅ =
𝑋𝑖
𝑛
=
106
15
= 7.067
Y̅ =
𝑌𝑖
𝑛
=
507
15
= 33.8
SSx= (𝑋𝑖 −X̅)2= 𝑋𝑖
2 − nX̄2 = 901.5 -
15((7.067)2)=152.363
SSY = (𝑌𝑖 − Ȳ)2 = 𝑌𝑖
2 − nȲ2= 17477-15((33.8)2) =
340.4
SSXY= (𝑋𝑖𝑌𝑖) − ( 𝑋𝑖)( 𝑌𝑖)/𝑛 = 3794 −
106 507
15
= 211.2
Sr. No. X Y XY X2
Y2
1 2 24 48 4 576
2 3 25 75 9 625
3 4 31 124 16 961
4 5 34 170 25 1156
5 11 40 440 121 1600
6 3 31 93 9 961
7 10 36 360 100 1296
8 8.5 36 306 72.25 1296
9 4 29 116 16 841
10 6.5 33 214.5 42.25 1089
11 8 37 296 64 1369
12 9.5 37 351.5 90.25 1369
13 11.5 39 448.5 132.3 1521
14 10.5 39 409.5 110.3 1521
15 9.5 36 342 90.25 1296
Total 106 507 3794 901.5 17477
Department of Mechanical Engineering

19. SOLUTION OF EXAMPLE CONTINUE..
ꞵ1=SSXY/SSX =
211.2
152.363
= 1.386
ꞵ0=Ȳ - ꞵ1X̅ = 33.8 – 1.386(7.067) = 24.004
The Regression Model Equation Y = ꞵ0 +ꞵ1X
Y = 24.004 + 1.386X
(i) When the investment is 13 Lakhs, the profit Y = 24.004 + 1.386(13)
=42.022 lakh
(ii) SSTotal = SSY = 340.4
SSR = ꞵ1*SSXY = (1.386)(211.2) = 292.723
SSe = SSTotal – SSR =340.4 – 292.723 = 47.677
3/16/2021 19
Department of Mechanical Engineering

20. EXAMPLE SOLUTION CONTINUE…
ANOVA TABLE FOR SIMPLE LINEAR REGRESSION
 Since F5%,1,13 = 4.67, regression is significant. That is the relation between X and Y
is significant.
3/16/2021 20
Source Sum of Square
Degree of
Freedom
Mean Square F0
Regression SSR=292.723 1 MSR=292.723
79.816
Error Sse=47.677 13 MSe=3.667
Total SSTotal 340.4 14
Department of Mechanical Engineering

21. EXAMPLE SOLUTION CONTINUE…
T-TEST FOR ꞵ1
The statistic to test the regression coefficient ꞵ1 is
t0 =
ꞵ1
𝑀𝑆𝑒
/𝑆𝑆𝑋
=
1.386
3.667
152.363
= 8.934
Since t0.025,13 = 2.160 the regression coefficient ꞵ1 is significant.
3/16/2021 21
Department of Mechanical Engineering

22. COEFFICIENT OF DETERMINATION R-SQUARE
 For this we have the following formula
R2 =
𝑆𝑆𝑅
𝑆𝑆𝑇𝑜𝑡𝑎𝑙
=
292.723
340.4
= 0.85994
 This mean that 85.994% change in the dependent variable occurred due to the
given explanatory variable, while rest of 14.006% may be caused by random error.
3/16/2021 22
Department of Mechanical Engineering

23. SIMPLE LINEAR REGRESSION
 The coefficients ꞵ0 & ꞵ1 and all other Regression result can also find from MS
Excel.
3/16/2021 23
Department of Mechanical Engineering