1. Predictive Analytics World
Weird Science:
How to Know Your Predictive
Discovery Is Not BS
Predictive Analytics World
May 2017
Eric Siegel, Ph.D.
Founder, Predictive Analytics World
Author, Predictive Analytics
2. Predictive Analytics World
If you torture the data long
enough, it will confess.
Ronald Coase
Professor of Economics
University of Chicago
“
”
There are three kinds of lies:
lies, damned lies, and statistics.
Benjamin Disraeli
“
”
20. Predictive Analytics World
The Data Effect:
Data is always predictive
Projects never fail due to lack of
patterns.
–Tom Khabaza
“
”
21. Predictive Analytics World
Technology that learns from experience to
predict the outcome/behavior of individuals
… in order to drive better decisions.
Predictive Analytics:
34 12 27 79 42 5934 1234 271234 79271234 42 5979271234 1234
(data)
^
22. Predictive Analytics World
Millions of decisions a day determine whom to call,
mail, approve, test, diagnose, warn, investigate,
incarcerate, set up on a date, and medicate.
52. Predictive Analytics World
70 hypotheses tested for baldness
Dirt
Broccoli
Bananas
Watching movies
Thinking hard
Raw eggs
Cooked eggs
Learning the tuba
Hip hop dancing
The Brady Bunch
Going to PAW
Exercise
Playing piano
Touch Bruce Willis
Vacation
Etc.
58. Predictive Analytics World
I am not saying that
Warren Buffett is not
skilled; only that a large
population of random
investors will almost
necessarily produce
someone with his track
records just by luck.
“
”–Nassim Nicholas
Taleb
59. Predictive Analytics World
... They [appear]
statistically significant,
but you have so many
lottery tickets when you
can run an analysis on a
[large data set] that
you’re going to have
some one-in-a-million
coincidences just by
chance alone.
“
”–Nate Silver
67. Predictive Analytics World
I got calls for years
asking me what the
current butter business
in Bangladesh was
looking like and I kept
saying, “Ya know, it was
a joke, it was a joke!” It’s
scary how few people
actually get that.
“
”–David Leinweber
82. Predictive Analytics World
Avoid the Vast Search Pitfall
Don't be fooled by randomness
Failsafe against cumulative risk
Account for vast search
86. Predictive Analytics World
Chapter 3: The Data Effect
Predictive Analytics: The Power to Predict Who Will
Click, Buy, Lie, or Die
Paper and Video:
Are Orange Cars Really not Lemons?
by John Elder and Ben Bullard, Elder Research, Inc.
www.elderresearch.com/orange-car
www.predictiveanalyticsworld.com/patimes/video-the-power-
and-peril-of-predictive-analytics/
Workshop: Predictive Modeling Methods
This Thursday
Instructor: John Elder
87. Predictive Analytics World
An unlimited amount of
computational resources
is like dynamite: If used
properly, it can move
mountains. Used
improperly, it can blow
up your garage or your
portfolio.
“
”
–David Leinweber
88. Predictive Analytics World
Why the Vast Search Pitfall
Is Still Prevalent
1) Simplicity can deceive
2) It’s elusive
3) It’s new
4) It's a buzzkill
5) Falsehoods don't look wrong
