L20_D.pdf

IPS
Introduction to Probability & Statistics
Lecture - 20
Prof. Pritam Ranjan

IPS
Regression
In statistics, regression analysis is a statistical process for
estimating the relationships between variables.
Pritam Ranjan / OM&QT 1

IPS
Motivating problem
1. Name: Name of cereal
2. mfr: Manufacturer of cereal (e.g., Kelloggs; Post; Quaker Oats, etc. )
3. type: cold or hot
4. calories: calories per serving
5. protein: grams of protein
6. fat: grams of fat
7. sodium: milligrams of sodium
8. fiber: grams of dietary fiber
9. carbo: grams of complex carbohydrates
10. sugars: grams of sugars
11. potass: milligrams of potassium
12. vitamins: vitamins and minerals % of FDA recommended
13. shelf: display shelf (1, 2, or 3, counting from the floor)
14. weight: weight in ounces of one serving
15. cups: number of cups in one serving
16. rating: a rating of the cereals

IPS
Simpler problem
Section G: red circle, Section H: blue traingle
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
20 40 60 80 100
30
40
50
60
70
80
90
Quiz 1
Midterm

IPS
Regression setup
• Variables types
Y - Response variable / Dependent variable
X - Predictor variable / Independent variable

IPS
Regression types
• Multiple regression model – One Y, multiple {X1, X2, ..., Xp}
• Simple regression model – One Y, one predictor X
• Simple linear regression model
• Non-linear regression model
� Spline regression
� Deep Learning models
� Gaussian regression
� non-parametric regression (smoothing)
• Generalized linear regression model
• Multivariate regression model

IPS
Simple linear regression (SLR) model
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
l
20 40 60 80 100
30
40
50
60
70
80
90
Quiz 1
Midterm

IPS
SLR- model statement
• Paired data: (Xi , Yi ), i = 1, 2, ..., n
• Model:
Yi = β0 + β1Xi + εi , for i = 1, 2, ..., n, and εi
iid
∼ N(0, σ2
)
OR
E(Y|X = Xi ) = β0+β1xi ,

IPS
SLR - model assumptions
• Errors have mean zero
E(εi ) = 0
• Errors have constant variance
Var(εi ) = σ2
• Errors are independent / uncorrelated
Corr(εi , εj ) = 0
• Relationship between Y and X is linear
E(Y|X = Xi ) = β0 + β1xi ,

IPS
Simple linear regression
• Model ﬁtting
⇒ estimating parameters
• Intuitively: what should the ﬁtted line look like?

IPS
Simple linear regression
• Model ﬁtting
→ how to estimate the parameters?

IPS
SLR - model ﬁtting
• Ordinary least squares (OLS) method
minimize Q =
n
�
i=1
ε2
i =
n
�
i=1
(Yi − β0 − β1Xi )2

IPS
SLR - parameter estimates
• Intercept
β̂0 = Ȳ − β̂1X̄
• Slope
β̂1 =
Sxy
Sxx
• Error-variance
σ̂2
=
1
n
n
�
i=1
(Yi − β̂0 − β̂1Xi )2
σ̂2
=
1
n − 2
n
�
i=1
(Yi − β̂0 − β̂1Xi )2

IPS
Example
• A pharmaceutical manufacturer wants to determine the concentration of a key
component of cough medicine that may be used without the drug’s causing
adverse side effects. As part of the analysis, a random sample of 45 patients is
administered doses of varying concentration (X), and the severity of side effects
(Y) is measured. Find the ﬁtted regression line.
x̄ = 88.9, ȳ = 165.3, Sxx = 2, 133.9, Sxy = 4, 502.53, Syy = 12, 500.

IPS
Inference

IPS
SLR - important distributions
• Error variable
εi
iid
∼ N(0, σ2
)
• Conditional distribution of response variable (NOT iid)
Y|(X = Xi ) ∼ N(β0 + β1Xi , σ2
)
• Slope parameter
β̂1 ∼ N
�
β1,
σ2
Sxx
�
• variance parameter
(n − 2)σ̂2
σ2
=
�n
i=1(Yi − β̂0 − β̂1Xi )2
σ2
∼ χ2
(n−2)

IPS
SLR - parameter of interest
• Slope parameter
β̂1 =
Sxy
Sxx
β̂1 ∼ N
�
β1,
σ2
Sxx
�
β1 ∈
�
β̂1 − tn−2,α/2
σ̂
Sx
, β̂1 + tn−2,α/2
σ̂
Sx
�
Reject H0 : β1 = 0 vs. Ha : β1 �= 0 if
�
�
�
�
�
β̂1 − 0
σ̂/Sx
�
�
�
�
�
≥ tn−2,α/2

IPS
SLR - parameter of interest
• Slope parameter β1 = 0 vs. β1 �= 0

IPS
Prediction

IPS
SLR - Prediction
• Two quantities of interest
(1) Estimate the regression line
E[Y|X = X0] = β0 + β1X0
(2) Predict a future observation
Y|(X = Xnew ) = β0 + β1Xnew + εnew

IPS
SLR - Prediction
• Conﬁdence interval - interval estimate for the regression line
[95% chance that your regression line will lie in this band]
• Prediction interval - interval estimate for predicting new observation
[95% chance that your new observation will lie in this band]

IPS
Model assessment

IPS
SLR - ANOVA
• Total variability - Sum of squares total
SSTo =
n
�
i=1
(Yi − Ȳ)2
df = n − 1
• Unexplained variability - Sum of squares error
SSE =
n
�
i=1
(Yi − Ŷi )2
df = n − k
• Variability explained by the model - Sum of squares regression
SSReg =
n
�
i=1
(Ŷi − Ȳ)2
df = k − 1

IPS
SLR - GOF (R2
)
• Goodness of ﬁt measures – R2
should be close to 1

IPS
Model Diagnostic

IPS
SLR - diagnostic check
• Is the relation between X and Y linear ?

IPS
• Is the error variance constant?

IPS
• Are the errors random?

IPS
• Are the errors normal?

IPS
• Are there any outliers?

IPS
Violations !!

IPS
Violation of Assumptions
• f(·) is non-linear (transformation /non-linear regression)
• Remove outliers (be super careful)
• Non-normal data (CI and testing via Bootstrapping)
• Non-constant variance (transformation / different model)
• εi ’s are correlated to each other
• Non-zero correlation between ε and X?
• Randomness in X is also present.

IPS
Next
• End-term
• Good luck !!

L20_D.pdf

Recommended

Recommended

More Related Content

Similar to L20_D.pdf

Similar to L20_D.pdf (20)

Recently uploaded

Recently uploaded (20)

L20_D.pdf