The document discusses the process of price optimization using data science techniques, emphasizing the importance of making sound assumptions and validating them through statistical testing. It outlines steps for harvesting big data, forecasting demand, and optimizing pricing while identifying potential issues with statistical assumptions related to linear regression models. The presentation concludes with a reminder to start with simple approaches and gradually build up complexity, reinforcing the necessity of testing assumptions in data analysis.

1. Assumptions:
Check yo self, before
you wreck yo self.
Erin Shellman @erinshellman
Seattle Software Craftsmanship
August 28, 2014
!

2. Assumptions:
Making an ass out of you
and me.
Erin Shellman @erinshellman
Seattle Software Craftsmanship
August 28, 2014
!

3. I’m Erin, and I’m a
data scientist.

4. How much should
this cost?

5. What about these?

6. …and when?
What about these?

7. Price optimization

8. Price optimization
1. Git yer
Big Data!

9. Price optimization
1. Git yer
Big Data!
2. Forecast
demand

10. Price optimization
1. Git yer
Big Data!
2. Forecast
demand
3. Optimize
price

11. Price optimization
4. Profit!!!!!
1.
Big Data!
2.
demand
3.
price

12. Price optimization
1. Git yer
Big Data!
2. Forecast
demand
3. Optimize
price
max
X
yi = !0 + !1xi + ✏i revenue

13. The key is a
good forecast.

15. Do the easiest thing
•Subset the data and focus on one category of
product.
• e.g. Alpine ski bindings.
• Prototype & validate in R.
Units Soldi = α + β1(pricei) + εi

16. Do the easiest thing
•Subset the data and focus on one category of
product.
• e.g. Alpine ski bindings.
• Prototype & validate in R.
Units Soldi = α + β1(pricei) + εi
Residual

17. Assumptions of SLR
•We assume that residuals:
1.Normal, with mean zero.
2.Are not autocorrelated.
3.Are unrelated to the predictors.

18. Checking assumptions is
hard
•…and boring!
•For statistical methods, assumption
testing traditionally relies on
visually inspecting plots (and lets
be real, most people don’t even
do that).

19. 40 60 80 100 120
0 500 1000 1500 2000 2500
Fitted values
Residuals
Residuals vs Fitted
117914
156
-3 -2 -1 0 1 2 3
0 2 4 6 8
Theoretical Quantiles
Standardized residuals
Normal Q-Q
194
171
156
40 60 80 100 120
0.0 0.5 1.0 1.5 2.0 2.5
Fitted values
Standardized residuals
Scale-Location
117914
156
0.00 0.01 0.02 0.03 0.04
0 2 4 6 8
Leverage
Standardized residuals
Cook's distance
1
0.5
Residuals vs Leverage
119741
109

20. OF all the practices you can
leverage to assist your
craftsmanship, you will get
the most benefit from testing.
!
Stephen Vance

21. test_that assumption!
context("Check assumptions of SLR")
!
test_that("The residuals are normally distributed", {
!
expect_that(shapiro.test(model_object$residuals)$p.value, is_more_than(0.05))
!
})
!
test_that("There is no autocorrelation", {
!
expect_that(lmtest::bgtest(model_object)$p.value, is_more_than(0.05))
!
})
!
test_that("The residuals are unrelated to the predictor", {
!
expect_that(cor(model_object$residuals, data$covariates), equals(0))
!
})
!

22. Tests pass!
> test_file("./tests/test_slr.R")
Check assumptions of SLR : [1] "units_sold ~ price"
...
!

23. Psych.
> test_file("./tests/test_slr.R")
Check assumptions of SLR : [1] "units_sold ~ price"
1..
!!
1. Failure(@test_slr.R#12): The residuals are normally distributed
------------------------
shapiro.test(model_object$residuals)$p.value not more than 0.05. Difference: 0.05
!

24. Linear? Eh.
•We assumed the
2500
functional form was
2000
linear, but there are
1500
several common forms
1000
that might better fit the
500
data. 0
100 200 300 400 500
Price ($)
Units Sold

25. Price ($)
Units Sold
Price ($)
Units Sold
Price ($)
Units Sold
Price ($)
Units Sold
Linear Log-log
Linear-log Log-linear

26. Price ($)
Units Sold
Price ($)
Units Sold
Price ($)
Units Sold
Price ($)
Units Sold
Linear response to change in price. Much more sensitive to change in price.
More gradual response to changes in price Sensitive initially, then gradual

30. # Automagically explore SLR with common functional forms
candidate_models = list(linear = 'units_sold ~ price',
loglog = 'log(units_sold + 1) ~ log(price + 1)',
linearlog = 'units_sold ~ log(price + 1)',
loglinear = 'log(units_sold + 1) ~ price')
!
run = function(candidate_models, input_data) {
forecasts = list()
test_input = data.frame(price = 0:1000)
!
# Forecast
for (model in candidate_models) {
test_environment = new.env()
!
# Generate the forecast
forecasts[[model]] = generate_forecast(model, input_data)
!
# Save off current value of things for testing
assign("model", forecasts[[model]], envir = test_environment)
assign("errors", forecasts[[model]]$residuals, envir = test_environment)
assign("covariate", input_data$price, envir = test_environment)
assign("label", model, envir = test_environment)
!
save(test_environment, file = 'env_to_test.Rda')
!
# Run assumption tests
test_file("./tests/test_slr.R")
!
#### OPTIMIZE PRICE!!! ####
opt_results = optimizer(forecasts[[model]], test_input)
!
# Multiply the predicted demand by the price for expected revenue
opt_results$expected_revenue = test_data$price * opt_results$predicted_units_sold
!
pdf(paste(model, “.pdf”, sep = ‘’))
plot_price(opt_results)
!
}
!
return(forecasts)
!
}

31. rut roh…
> run(candidate_models, slr_data)
Check assumptions of SLR : [1] "units_sold ~ price"
1..
!!
1. Failure(@test_slr.R#12): The residuals are normally distributed ---------------------------------
shapiro.test(linear$residuals)$p.value not more than 0.05. Difference: 0.05
!
Check assumptions of SLR : [1] "log(units_sold + 1) ~ log(price + 1)"
1.2
!!
1. Failure(@test_slr.R#12): The residuals are normally distributed ---------------------------------
shapiro.test(linear$residuals)$p.value not more than 0.05. Difference: 0.05
!
2. Failure(@test_slr.R#24): The residuals are unrelated to the predictor ---------------------------
cor(test_environment$errors, test_environment$covariate) not equal to 0
Mean absolute difference: 0.05545615
!
Check assumptions of SLR : [1] "units_sold ~ log(price + 1)"
1.2
!!
1. Failure(@test_slr.R#12): The residuals are normally distributed ---------------------------------
shapiro.test(linear$residuals)$p.value not more than 0.05. Difference: 0.05
!
2. Failure(@test_slr.R#24): The residuals are unrelated to the predictor ---------------------------
cor(test_environment$errors, test_environment$covariate) not equal to 0
Mean absolute difference: 0.04201906
!
Check assumptions of SLR : [1] "log(units_sold + 1) ~ price"
1..
!!
1. Failure(@test_slr.R#12): The residuals are normally distributed ---------------------------------
shapiro.test(linear$residuals)$p.value not more than 0.05. Difference: 0.05

32. 20000
15000
10000
5000
0
Linear Log-log
0 250 500 750 1000
Price ($)
Expected Revenue
15000
10000
5000
0
0 250 500 750 1000
Price ($)
Expected Revenue
Linear-log Log-linear
6000
4000
2000
0
0 250 500 750 1000
Price ($)
Expected Revenue
60000
40000
20000
0
0 250 500 750 1000
Price ($)
Expected Revenue

33. 20000
15000
10000
5000
Optimal Price = $322
0
Linear Log-log
0 250 500 750 1000
Price ($)
Expected Revenue
15000
10000
5000
0
0 250 500 750 1000
Price ($)
Expected Revenue
Linear-log Log-linear
6000
4000
2000
0
0 250 500 750 1000
Price ($)
Expected Revenue
60000
40000
20000
0
0 250 500 750 1000
Price ($)
Expected Revenue
Optimal Price > $1000
Optimal Price = $∞
Optimal Price = $779

35. Mean = 185
40
30
20
10
0
100 200 300 400
Price ($)
Counts

36. In conclusion, these
forecasts suck.
We are just
getting
warmed up!

37. Beginner-Intermediate Intermediate-Advanced Advanced-Expert
2000
1500
1000
500
0
0 100 200 300 400 5000 100 200 300 400 5000 100 200 300 400 500
Price ($)
Units Sold

38. 2011-06-01 2011-10-01 2012-02-01 2012-06-01 2012-10-01 2013-02-01 2013-06-01 2013-10-01 2014-02-01
Date
Units Sold

39. 2011-06-01 2011-10-01 2012-02-01 2012-06-01 2012-10-01 2013-02-01 2013-06-01 2013-10-01 2014-02-01
Date
Units Sold
TIME?!

40. Try something a little
smarter
Units Soldi = α + β1(pricei) + β2(abilityi) + β3(monthi) + εi

41. Beginner-Intermediate Intermediate-Advanced Advanced-Expert
15000
10000
5000
0
15000
10000
5000
0
15000
10000
5000
0
15000
10000
5000
0
15000
10000
5000
0
15000
10000
5000
0
15000
10000
5000
0
15000
10000
5000
0
15000
10000
5000
0
15000
10000
5000
0
15000
10000
5000
0
15000
10000
5000
0
1 2 3 4 5 6 7 8 9 10 11 12
0 250 500 750 1000 0 250 500 750 1000 0 250 500 750 1000
Price ($)
Expected Revenue

42. Yeah, but who cares?
•Do we need to throw everything out
just because some assumptions are
invalidated?
•What is our goal?
•Is it still better than what we did
previously?

43. Wrap it up.
1. Do the easiest thing first, and do it well.
It’s how you’re going to learn the domain,
and it’s your benchmark for improvement.
2. Test your assumptions, and invest time in
building the tools needed to do that
effectively.
3. Be cool, stay in school.

44. Thanks bros!!
Nathan Decker, Brian Pratt & the Evo crew 
Jason Gowans & Bryan Mayer 
Elissa “Downtown” Brown, forecasting genius 
John Foreman, MailChimp 
#nordstromdatalab 

45. Click-bait!
1. Data Carpentry: http://mimno.infosci.cornell.edu/b/articles/carpentry/
2. Getting started with testthat. http://journal.r-project.org/archive/2011-1/
RJournal_2011-1_Wickham.pdf
3. Clean Code: http://www.amazon.com/Clean-Code-Handbook-Software-
Craftsmanship/dp/0132350882/
4. Quality Code: http://www.amazon.com/Quality-Code-Software-Principles-
Practices/dp/0321832981
5. Revenue Management: http://www.amazon.com/Practice-Management-
International-Operations-Research/dp/0387243763/
6. Pricing and Revenue Optimization: http://www.amazon.com/Pricing-Revenue-
Optimization-Robert-Phillips-ebook/dp/B005JTDOVE/
7. Original G, Rob Hyndman: https://www.otexts.org/fpp and http://
robjhyndman.com/hyndsight/