Linear regression

1. Linear regression

2. What is a Math/Stats Model
Describe relationships between variables
Deterministic models (no randomness)
Probabilistic models (with randomness)

3. Deterministic Models
Hypothesize exact relationships
Suitable when prediction error is negligible
Example: Gravity force: F = Gm m /d
1 2
2

4. Probabilistic Models
Hypothesize two components of the relationship
Deterministic
Random error
Example: Systolic blood pressure of newborns: p = 6d + ϵ
Random error may be due to other factors (e.g. birth weight)

5. Regression Model
Model relationship between one dependent variable and one or several explanatory variable(s)
bug = α * code size + β * prior bugs + γ * changes + ϵ
Used mainly for prediction and estimation

6. Regression Modeling Steps
1. Hypothesize deterministic component
2. Specify probability distribution of random error term
3. Evaluate fitted model
4. Use model for prediction and estimation

7. Model Specification
Specifying the deterministic component
1. Define the dependent variable and independent variable
2. Hypothesize nature of relationship
Functional form (e.g. linear or non‑linear)
Expected Effects (i.e., signs of coefficients)
Interactions between variables

8. Linear Regression Model
Relationship between variables is a linear function
Y = aX + b + ϵ

9. Estimating Parameters
Compute model parameters that best fit data

10. Example: Cow's food intake and milk
Food (lb) Milk yield (lb)
4 3.0
6 5.5
10 6.5
12 9.0

11. Least Squares Method
Best Fit means minimized sum of squared errors (SSE)

12. Interpretation of Coefficients
Slope: Estimated Y changes by for each 1 unit increase in X
Intercept: Average value of Y when X = 0

13. Explanatory and predictive power
R is the measurement of goodness‑of‑fit, i.e., How the model fits to all training data
R = 1 − where Y is the actual dependent variable and is the fitted
R is also called a measurement of explanatory power, i.e. how well the model explains the
data it is trained on
Predictive power indicates how well the model predicts the new data (data not used for
training, also called testing data)
MAE = mean(∣ − Y ∣) where where Y is the actual dependent variable and is the
predicted on testing data
2
2
var(Y )
var( −Y )
Y
^
Y
^
2
Ȳ Ȳ

14. Cross‑validation
Is used to compute predictive power when only a dataset is available:
1. Divide dataset into two subsets: training and testing data
2. Train the model on training data and make prediction for testing data
3. Repeat many times
4. Compute the final mean absolute error