- Jeff Leek taught a course called "Data Analysis" through Coursera that had 7,000 students enrolled, far more than expected. - Leek discussed different theoretical models for how data analysis is performed and how inferences can be affected by choices in the data analysis process. - Experiments were discussed as a way to make causal claims about relationships, rather than just inferential claims, with examples given about studies on smoking and lung cancer.

2. Evidence
based
data
analysis @jtleek

3. Data science
as a
Science
(DSaaS) @jtleek

9. “Data science is as much
art as it is science.”

11. Wouldn’t it be amazing if we got
2,000 people to learn statistics!
“
”-Jeff Leek
7/17/12

12. date: 7/19/12
from: jtleek@gmail.com
Roger let me know you gave him a
ballpark figure for the number of
students registered for his course
"Computing for Data Analysis”. Could
you give me an idea of how many have
registered for my course "Data
Analysis?”

13. date: 7/19/12
from: pangwei@coursera.org
Hi Jeff,
7,000 students! It's pretty awesome.
(You'll be able to check this out yourself
next week, once the class sites are up.)

14. date: 7/19/12
from: rdpeng@gmail.com
You are f**ed.
-roger

15. 9 classes
1 month long
Always open

16. Data Science Specialization
Total Enrollments: 3,815,890
Total Completions: 409,712
Genomic Data Science Specialization
Total Enrollments: 173,495
Total Completions: 10,826
Executive Data Science
Specialization
Total Enrollments: 62,076

18. A theoretical model
Data

19. A theoretical model
Data

20. Y = some outcome
X = some covariate
D = (X,Y)
lm(Y ~ X)

21. Y = some outcome
X = some covariate
D = (X,Y)
lm(Y ~ X)
Leek and Peng, Nature 2015

22. F0
Ul
F0(S)
Ul
F0(Y)
Fithian, Sun and Taylor arXiv 2015

23. σ-algebra
“what we know”
F0
Ul
F0(S)
Ul
F0(Y)

24. “we’ve done nothing”
F0
Ul
F0(S)
Ul
F0(Y)

25. “we did model selection”
F0
Ul
F0(S)
Ul
F0(Y)

26. “we looked at all the data”
F0
Ul
F0(S)
Ul
F0(Y)

27. E[β |F0]
≠
E[β |F0(S)]

28. Population
Question
Hypothesis
Experimental Design
Experimentor
Data
Analysis Plan
Analyst
Code
Estimate
Claim Patil, Peng and Leek biorXiv 2016

29. Population
Question
Hypothesis
Experimental Design
Experimentor
Data
Analysis Plan
Analyst
Code
Estimate
Claim Patil, Peng and Leek biorXiv 2016
F0
Ul
F0(1P,Q(H))
F0(1ED(E))
F0(1ED;E(D))
F0(1AP;A(C))
F0(1C(A*))
UlUlUlUl

30. Population
Question
Hypothesis
Experimental Design
Experimentor
Data
Analysis Plan
Analyst
Code
Estimate
Claim Patil, Peng and Leek biorXiv 2016
F0
Ul
F0(1P,Q(H))
F0(1ED(E))
F0(1ED;E(D))
F0(1AP;A(C))
F0(1C(A*))
UlUlUlUl

31. A theoretical model
Data

32. Slide courtesy Hadley Wickham

33. Who?
What?
When?
Why?
Where?
How? Slide courtesy Hadley Wickham

34. Who?
What?
When?
Why?
Where?
How? Where Ingo is working

35. Who?
What?
When?
Why?
Where?
How? Slide courtesy Hadley Wickham
Base R
Lassodplyr
googlesheets
ppt

36. Who?
What?
When?
Why?
Where?
How? Slide courtesy Hadley Wickham
Bad life choices?
Sparsity!
David Robinson
told me
Spreadsheets 
Hedgemony

37. Cleveland and McGill JASA 1984

39. Leek & Peng 2015 PNAS

40. Experiment

41. Leek and Peng, Science 2015

42. Population
Question
Hypothesis
Experimental Design
Experimentor
Data
Analysis Plan
Analyst
Code
Estimate
Claim
E[S| F0(1c(W))

43. We take a random sample of individuals in a
population and identify whether they smoke
and if they have cancer. We observe that there
is a strong relationship between whether a
person in the sample smoked or whether they
have lung cancer. We claim that smoking is
related to lung cancer in the larger population.

44. 79% 17%
Inferential
vs
Causal
n=47,141

45. We take a random sample of individuals in a
population and identify whether they smoke
and if they have cancer. We observe that there
is a strong relationship between whether a
person in the sample smoked or whether they
have lung cancer. We claim that smoking is
related to lung cancer in the larger population.
We explain we think that the reason for this
relationship is because cigarette smoke
contains known carcinogens such as benzene,
which make cells in lungs become cancerous.

46. 65% 32 %
Inferential
vs
Causal
n=47,141

47. Experiment

48. Population
Question
Hypothesis
Experimental Design
Experimentor
Data
Analysis Plan
Analyst
Code
Estimate
Claim
E[Est| F0(1c(A))

51. 69% vs 40%
n=1,985

52. Experiment

54. E[Claim | F0(1set(base)(A))]
-
E[Claim | F0(1set(ggplot2)(A))]
Population
Question
Hypothesis
Experimental Design
Experimentor
Data
Analysis Plan
Analyst
Code
Estimate
Claim

55. 1. Make a plot that answers the question: what is the
relationship between mean covered charges
(Average.Covered.Charges) and mean total payments
(Average.Total.Payments) in New York?
2. Make a plot (possibly multi-panel) that answers the
question: how does the relationship between mean
covered charges (Average.Covered.Charges) and mean
total payments (Average.Total.Payments) vary by
medical condition (DRG.Definition) and the state in which
care was received (Provider.State)?
Use only the [ggplot2/base R] graphics system (not
base R or lattice) to make your figure.

56. “Does the plot clearly show the
relationship between mean covered
charges (Average.Covered.Charges)
and mean total payments
(Average.Total.Payments) in New
York?”
G: 5/22 (23%) vs. B: 5/12
(42%)

57. “Does the plot clearly show the relationship
between mean covered charges
(Average.Covered.Charges) and mean total
payments (Average.Total.Payments) vary by
medical condition (DRG.Definition) and the state
in which care was received (Provider.State)?”
G: 7/22 (32%) vs. B: 5/12 (42%)

58. “Is the plot visually pleasing?”
G: 21/22 (95%) vs. B: 10/12 (83%)
G: 20/22 (91%) vs. B: 8/12 (67%)

59. “Do the plot text and labels use full
words instead of abbreviations?”
G: 21/22 (95%) vs. B: 12/12 (100%)
G: 11/22 (50%) vs. B: 5/12 (42%)

60. A theoretical model
Data

61. Data science
as a
Science
(DSaaS) @jtleek