835 keynote siegel
•
2 likes•304 views
This document discusses how to avoid making false conclusions from predictive analytics due to issues related to analyzing large datasets. It notes that analyzing large datasets increases the risk of finding patterns that are not truly predictive due to random chance. This is known as the "vast search pitfall". The document provides examples of studies that have fallen victim to this, and emphasizes the importance of accounting for dataset size and multiple hypothesis testing. It recommends strategies like using a test set of held-out data and adjusting significance thresholds to avoid being fooled by randomness in large datasets.
Recommended
Recommended
More Related Content
Similar to 835 keynote siegel
Similar to 835 keynote siegel (20)
More from Rising Media, Inc.
More from Rising Media, Inc. (20)
Recently uploaded
Recently uploaded (20)
835 keynote siegel
- 1. Predictive Analytics World Weird Science: How to Know Your Predictive Discovery Is Not BS Predictive Analytics World May 2017 Eric Siegel, Ph.D. Founder, Predictive Analytics World Author, Predictive Analytics
- 2. Predictive Analytics World If you torture the data long enough, it will confess. Ronald Coase Professor of Economics University of Chicago “ ” There are three kinds of lies: lies, damned lies, and statistics. Benjamin Disraeli “ ”
- 3. Predictive Analytics World When is bigger data more dangerous
- 6. Predictive Analytics Worldwww.ThePredictionBook.com Intro - Concepts Chapter 1 - Deployment Chapter 2 - Ethics Chapter 3 - Data Chapter 4 - Modeling Chapter 5 - Ensembles Chapter 6 - Watson Chapter 7 - Influence
- 7. Predictive Analytics World Chapter 3: The Data Effect
- 8. Predictive Analytics World Outline 1. Value – the sexiest job 2. Caution – orange lemons 3. False conclusions – cumulative risk 4. Comedy – humorous examples 5. Safeguards – ensuring sound results
- 20. Predictive Analytics World The Data Effect: Data is always predictive Projects never fail due to lack of patterns. –Tom Khabaza “ ”
- 21. Predictive Analytics World Technology that learns from experience to predict the outcome/behavior of individuals … in order to drive better decisions. Predictive Analytics: 34 12 27 79 42 5934 1234 271234 79271234 42 5979271234 1234 (data) ^
- 22. Predictive Analytics World Millions of decisions a day determine whom to call, mail, approve, test, diagnose, warn, investigate, incarcerate, set up on a date, and medicate.
- 23. Predictive Analytics World Predictive analytics is the Information Age's latest evolutionary step.
- 24. Predictive Analytics World A Cautionary Tale
- 25. Predictive Analytics World An orange used car is least likely to be a lemon. “ ” 2012
- 33. Predictive Analytics World p < 1% if orange cars were no different, there’d be a < 1% chance we’d see that extreme a result anyway
- 34. Predictive Analytics World p < 0.0068
- 35. Predictive Analytics World BAD SCIENCE
- 36. Predictive Analytics World The trouble with the world is that the stupid are cocksure and the intelligent are full of doubt. –Bertrand Russell “ ”
- 37. Predictive Analytics World An Insightful Story
- 40. Predictive Analytics World 70 spins...
- 45. Predictive Analytics World The Scientific Method
- 46. Predictive Analytics World Predictive analytics is the automation of scientific research.
- 48. Predictive Analytics World “Less than 1% chance woulda shown by chance if it weren’t a real effect.”
- 49. Predictive Analytics World 70 hypotheses...
- 50. Predictive Analytics World “ ” un-
- 51. Predictive Analytics World It is probable that improbable things will happen. –Aristotle “ ”
- 52. Predictive Analytics World 70 hypotheses tested for baldness Dirt Broccoli Bananas Watching movies Thinking hard Raw eggs Cooked eggs Learning the tuba Hip hop dancing The Brady Bunch Going to PAW Exercise Playing piano Touch Bruce Willis Vacation Etc.
- 54. Predictive Analytics World Make Model Year Size Color
- 55. Predictive Analytics World Vast search John Elder, CEO, Elder Research, Inc. AKA: Multiple comparisons trap Multiple hypothesis testing Researcher degrees of freedom over-search (akin to overfit) Look-elsewhere effect The garden of forking paths Fishing expedition Cherry-picking findings Data dredging Significance chasing P-hacking
- 56. Predictive Analytics World bigger data is more hazardous wider___
- 57. Predictive Analytics World Make Model Year Size Color
- 58. Predictive Analytics World I am not saying that Warren Buffett is not skilled; only that a large population of random investors will almost necessarily produce someone with his track records just by luck. “ ”–Nassim Nicholas Taleb
- 59. Predictive Analytics World ... They [appear] statistically significant, but you have so many lottery tickets when you can run an analysis on a [large data set] that you’re going to have some one-in-a-million coincidences just by chance alone. “ ”–Nate Silver
- 61. Predictive Analytics World In China when you’re one in a million, there are 1,300 people just like you. –Bill Gates “ ”
- 63. Predictive Analytics World XKCD, by Randall Munroe
- 66. Predictive Analytics World Butter in Bangladesh
- 67. Predictive Analytics World I got calls for years asking me what the current butter business in Bangladesh was looking like and I kept saying, “Ya know, it was a joke, it was a joke!” It’s scary how few people actually get that. “ ”–David Leinweber
- 71. Predictive Analytics World 72,983 cars – 8,976 lemons 415 orange cars – 34 lemons
- 72. Predictive Analytics World 72,983 cars – 8,976 lemons 415 orange cars – 34 lemons p < 0.0068
- 74. Predictive Analytics World Target Shuffling
- 76. Predictive Analytics World Bad news: The vast search pitfall is prevalent.
- 81. Predictive Analytics World Good news: You can avoid the vast search pitfall.
- 82. Predictive Analytics World Avoid the Vast Search Pitfall Don't be fooled by randomness Failsafe against cumulative risk Account for vast search
- 84. Predictive Analytics World 1) Bigger Longer Data
- 85. Predictive Analytics World 2) Employ a Test Set
- 86. Predictive Analytics World Chapter 3: The Data Effect Predictive Analytics: The Power to Predict Who Will Click, Buy, Lie, or Die Paper and Video: Are Orange Cars Really not Lemons? by John Elder and Ben Bullard, Elder Research, Inc. www.elderresearch.com/orange-car www.predictiveanalyticsworld.com/patimes/video-the-power- and-peril-of-predictive-analytics/ Workshop: Predictive Modeling Methods This Thursday Instructor: John Elder
- 87. Predictive Analytics World An unlimited amount of computational resources is like dynamite: If used properly, it can move mountains. Used improperly, it can blow up your garage or your portfolio. “ ” –David Leinweber
- 88. Predictive Analytics World Why the Vast Search Pitfall Is Still Prevalent 1) Simplicity can deceive 2) It’s elusive 3) It’s new 4) It's a buzzkill 5) Falsehoods don't look wrong
- 89. Predictive Analytics World To Learn More: The Predictive Analytics Guide www.pawcon.com/guide
- 91. Eric Siegel, Ph.D. Program Chair Predictive Analytics World