The line that minimizes the residual
error, or point distance from the line
which is why we refer to the regression line
as the least squares error regression line
We determine this with a linear
regression to determine the y-intercept
and the slope of the line that minimizes
the error residuals
Use the lm function in R to create linear models
A regression on one variable is known as a simple
• relationship of predictors to predicted variables
• the variance of error terms is constant
• minimal to no outliers in the data (high or low y
in response to x)
• minimal to no leverage points in the data (high
or low x relative to the data)
• no collinearity among predictor variables
• predictors are additive to reliability of model
(no interaction effects)
Bonds dataset from
“A Modern Approach to Regression with R” Sheather, Simon. 2009.
Consider the following dataset of bond prices
bonds.dat = read.csv("http://www.stat.tamu.edu/~sheather/book/docs/datasets/bonds.txt", sep='t')
Data analysis is the art of asking
questions of the data and searching
What questions should we ask?
• Is Bid Price a function of Coupon
Rate or vice versa?
• What type of relationship does Bid
Price appear to have with Coupon
• Is there a formula we could use to
predict Bid Price based on Coupon
Linear Model of Bond Prices
as a function of Coupon Rates
• Is this line a good or
bad ﬁt with the data?
• Why or why not?
• Can we improve the
Know your data and know
how your models work!
• Why would outliers skew
the linear regression
• What should we do
• Why do these outliers
What are the outliers?
A Flower bond is a U.S treasury bond recoverable
before maturity upon payment or fulfilling a condition, if
used to settle federal estate taxes. When flower bonds
are surrendered in payment of taxes, and accepted as
such, that constitutes payment of those taxes for statute
of limitations and statutory interest purposes.
Adjusted Bond Model
• After removing the
outliers the model looks
• But how do you know
that it’s a better model?
Evaluating the two
Key measures are p-value,
Residual Std Error (RSE), and R2
Residual Sum of Squares (RSS)
“In statistics, the residual sum of squares (RSS), also known as the
sum of squared residuals (SSR) or the sum of squared errors of
prediction (SSE), is the sum of the squares of residuals
(deviations predicted from actual empirical values of data).”
What is the best model?
The most accurate model on the training data is
always the model with the most predictor
variables (p) and the lowest residual sum of
squares (RSS)…but what is the best model?
The best model is…
The best model is the simplest model
with the most predictive power on the
entire data population while staying
within your resource constraints.
Bonus: Easy and tidy with
the broom package
From the broom vignette:
The broom package takes the messy output of built-in functions in R, such as lm,
nls, or t.test, and turns them into tidy data frames.
This package provides three S3 methods that do three distinct kinds of tidying.
• tidy: constructs a data frame that summarizes the model's statistical ﬁndings.
This includes coefﬁcients and p-values for each term in a regression, per-cluster
information in clustering applications, or per-test information for multtest
• augment: add columns to the original data that was modeled. This includes
predictions, residuals, and cluster assignments.
• glance: construct a concise one-row summary of the model. This typically
contains values such as R
and residual standard error that are
computed once for the entire model.