A set of slides from my closing keynote at DamnData. I go over the concept of the Data Scientific Method, the skills required for a data scientists from hacking, to maths and stats, to expertise to business knowledge. I also talk about some ideas we worked on, some tools we use, some technologies and the most important part, the questioning of the data.

1. Data Science?!
what even...

2. David Coallier
@davidcoallier

3. Data Scientist
Engine Yard

5. And I cook..
A lot.

6. (n-1) items

7. Adapting.

8. Feedback.

9. Indifference.

10. Young mathematically
inclined minds

11. Young mathematically inclined minds
We knew everything.

12. First Bad Assumption.

13. So we asked “experts”.

14. Bad Ingredients

15. Bad Data

16. Tasted like sh*t

20. From Our Results
We had questions.

21. Found Expertise
Not Online.

22. Data Scientific
Method

23. Find a Question
Your Hypothesis

24. Current Data
What do you have?

25. Features & Tests
Try it.

26. Analyse Results
Won’t be pretty.

27. Conversation
Framed. By. Data.

28. But....

29. Good Discussions
Imply good data scientists

30. Hacking Skills

31. Hacking Skills
Maths &
Stats

32. Hacking Skills
Expertise
Maths &
Stats

33. Hacking Skills
Machine
Learning
Danger
Zone!!!
Expertise
Research
Maths &
Stats

34. Hacking Skills
Data
Science
Expertise
Maths &
Stats

35. Hacking Skills
Danger
Zone!!!
Machine
Learning
Data
Science
Maths &
Stats
Expertise
Research

36. Business
Don’t need an MBA

37. In other words.

38. 1. Hacking
2. Maths & Stats
3. Expertise

39. Apply Method
Data Scientific

40. 1. Question
2. Current Data
3. Features/Tests
4. Analyse
5. Converse

41. Find a Question
Let’s imagine Github

42. Upgrade Repos
Affect users as little as possible

43. import csv
content = csv.read('repo1.csv')

44. λ e
f (k; λ ) =
k!
k −k
for k >= 0

46. Converse
Present Findings

47. Iterate
Commits aren’t key.

48. KPIs are key
Indicators from experience

49. Questions
Super Important.

50. Just test it..

54. We are Human.
Emotional Connection

55. What next?
Second Hypothesis.

57. Focus on Data
Relevant to your KPIs.

59. Data gives you the what
Humans give you the why

60. Turn Information

61. Into
Actionable Insight

64. Create Discussions
Introspection Engines

65. Seeing, Feeling it
The brain sees.

66. Not regressions

67. Not p-values

68. Not slopes

69. Not F-statistics

70. Not coefficients

73. Another Example
Fraud Engine

74. Features
Fraud Engine

75. Clusters
User Types

76. Machine Learning
Historical Analysis

78. Decision
Report as Fraudulent

79. Fact-Based
Decision Failing

80. Fact-Based
Decision Making

81. Action
Measure
Knowledge
Analysis

82. Failed.
Noetic Intelligence

83. Action
Measure
Knowledge
Analysis

84. Action
Measure
Knowledge
Analysis

85. Offering
Missing Feature

86. Toolbox
What do we use?

87. R
Modeling, Testing, Prototyping

88. RStudio
The IDE

89. lubridate
and zoo
Dealing with Dates...

90. yy/mm/dd
mm/dd/yy
YYYY-mm-dd HH:MM:ss TZ
yy-mm-dd
1363784094.513425
yy/mm
different timezone

91. reshape2
Reshape your Data

92. ggplot2
Visualise your Data

93. RCurl, RJSONIO
Find more Data

94. HMisc
Miscellaneous useful functions

95. forecast
Can you guess?

96. garch
Generalized Autoregressive
Conditional Heteroskedasticity

97. quantmod
Statistical Financial Trading

99. getSymbols('AAPL')
barChart(AAPL)
addMACD()

100. xts
Extensible Time Series

101. igraph
Study Networks

102. maptools
Read & View Maps

103. map('state', region = c(row.names(USArrests)), col=cm.colors(16, 1)[floor(USArrests$Rape/max(USArrests$Rape)*28)], fill=T)

104. Python
Scientific Computing

105. SciPy
http://www.scipy.org

106. scipy.stats

107. scipy.stats
Descriptive Statistics

108. from scipy.stats import
describe
s = [1,2,1,3,4,5]
print describe(s)

109. scipy.stats
Probability Distributions

110. Example
Poisson Distribution

111. λ e
f (k; λ ) =
k!
k −k
for k >= 0

112. import scipy.stats.poisson
p = poisson.pmf([1,2,3,4,1,2,3], 2)

113. print p.mean()
print p.sum()
...

114. NumPy
http://www.numpy.org/

115. NumPy
Linear Algebra

116. ⎛ 1 0 ⎞
⎜ 0 1 ⎟
⎝
⎠

117. import numpy as np
x = np.array([ [1, 0], [0, 1] ])
vec, val = np.linalg.eig(x)
np.linalg.eigvals(x)

118. >>> np.linalg.eig(x)
(
array([ 1., 1.]),
array([
[ 1., 0.],
[ 0., 1.]
])
)

119. Matplotlib
Python Plotting

120. statsmodels
Advanced Statistics Modeling

121. NLTK
Natural Language Tool Kit

122. scikit-learn
Machine Learning

123. from sklearn import tree
X = [[0, 0], [1, 1]]
Y = [0, 1]
clf = tree.DecisionTreeClassifier()
clf = clf.fit(X, Y)
clf.predict([[2., 2.]])
>>> array([1])

124. PyBrain
... Machine Learning

125. PyMC
Bayesian Inference

126. Pattern
Web Mining for Python

127. NetworkX
Study Networks

128. MILK: Machine Learning

129. Pandas
easy-to-use data structures

130. from pandas import *
x = DataFrame([
{"age": 26},
{"age": 19},
{"age": 21},
{"age": 18}
])
print x[x['age'] > 20].count()
print x[x['age'] > 20].mean()

131. Python vs R?
Different Purposes

132. Sto
rage

133. Oppose
“big” Data

134. Hadoop

135. Had - oops

136. Riak
Key-Value Buckets

137. CouchDB
Document Database

138. Redis
In-Memory Database

139. Cube
Time-series Database

140. PgSQL
Quite Extensively

141. Visualisation

142. Right Now
The rule of 3

143. Engineer
Report One

144. Mid-Level Mgr
Report Two

145. Board Level
Report Three

146. The Future
Discoverable Insight

147. d3.js
Data-Driven Documents

148. The Future
Discoverable Insight

151. Dashing
Elegant Dashboards

153. Edward Tufte
Go read his books.

154. Dogfooding
Data Scientific Method

155. Original Question
What is Data Science?

156. Back to you
For questioning