Analytics is what will drive big data adoption

1. Big Data is Not About the Data!
Gary King1
Institute for Quantitative Social Science
Harvard University
Talk at the History of Evidence class, Harvard Law School, 11/17/2014
1GaryKing.org
1/10

2. The Value in Big Data: the Analytics
2/10

3. The Value in Big Data: the Analytics
Data:
2/10

4. The Value in Big Data: the Analytics
Data:
easy to come by; often a free byproduct of IT improvements
2/10

5. The Value in Big Data: the Analytics
Data:
easy to come by; often a free byproduct of IT improvements
becoming commoditized
2/10

6. The Value in Big Data: the Analytics
Data:
easy to come by; often a free byproduct of IT improvements
becoming commoditized
Ignore it every institution will have more every year
2/10

7. The Value in Big Data: the Analytics
Data:
easy to come by; often a free byproduct of IT improvements
becoming commoditized
Ignore it every institution will have more every year
With a bit of eort: huge data production increases
2/10

8. The Value in Big Data: the Analytics
Data:
easy to come by; often a free byproduct of IT improvements
becoming commoditized
Ignore it every institution will have more every year
With a bit of eort: huge data production increases
Where the Value is: the Analytics
2/10

9. The Value in Big Data: the Analytics
Data:
easy to come by; often a free byproduct of IT improvements
becoming commoditized
Ignore it every institution will have more every year
With a bit of eort: huge data production increases
Where the Value is: the Analytics
Output can be highly customized
2/10

10. The Value in Big Data: the Analytics
Data:
easy to come by; often a free byproduct of IT improvements
becoming commoditized
Ignore it every institution will have more every year
With a bit of eort: huge data production increases
Where the Value is: the Analytics
Output can be highly customized
Moore's Law (doubling speed/power every 18 months)
2/10

11. The Value in Big Data: the Analytics
Data:
easy to come by; often a free byproduct of IT improvements
becoming commoditized
Ignore it every institution will have more every year
With a bit of eort: huge data production increases
Where the Value is: the Analytics
Output can be highly customized
Moore's Law (doubling speed/power every 18 months)
v. Our Students (1000x speed increase in 1 day)
2/10

12. The Value in Big Data: the Analytics
Data:
easy to come by; often a free byproduct of IT improvements
becoming commoditized
Ignore it every institution will have more every year
With a bit of eort: huge data production increases
Where the Value is: the Analytics
Output can be highly customized
Moore's Law (doubling speed/power every 18 months)
v. Our Students (1000x speed increase in 1 day)
$2M computer v. 2 hours of algorithm design
2/10

13. The Value in Big Data: the Analytics
Data:
easy to come by; often a free byproduct of IT improvements
becoming commoditized
Ignore it every institution will have more every year
With a bit of eort: huge data production increases
Where the Value is: the Analytics
Output can be highly customized
Moore's Law (doubling speed/power every 18 months)
v. Our Students (1000x speed increase in 1 day)
$2M computer v. 2 hours of algorithm design
Low cost; little infrastructure; mostly human capital needed
2/10

14. The Value in Big Data: the Analytics
Data:
easy to come by; often a free byproduct of IT improvements
becoming commoditized
Ignore it every institution will have more every year
With a bit of eort: huge data production increases
Where the Value is: the Analytics
Output can be highly customized
Moore's Law (doubling speed/power every 18 months)
v. Our Students (1000x speed increase in 1 day)
$2M computer v. 2 hours of algorithm design
Low cost; little infrastructure; mostly human capital needed
Innovative analytics: enormously better than o-the-shelf
2/10

15. Examples of what's now possible
3/10

16. Examples of what's now possible
Opinions of activists:
3/10

17. Examples of what's now possible
Opinions of activists: A few thousand interviews
3/10

18. Examples of what's now possible
Opinions of activists: A few thousand interviews billions of
political opinions in social media posts (1B every 2 Days)
3/10

19. Examples of what's now possible
Opinions of activists: A few thousand interviews billions of
political opinions in social media posts (1B every 2 Days)
Exercise:
3/10

20. Examples of what's now possible
Opinions of activists: A few thousand interviews billions of
political opinions in social media posts (1B every 2 Days)
Exercise: A survey: How many times did you exercise last
week?
3/10

21. Examples of what's now possible
Opinions of activists: A few thousand interviews billions of
political opinions in social media posts (1B every 2 Days)
Exercise: A survey: How many times did you exercise last
week? 500K people carrying cell phones with
accelerometers
3/10

22. Examples of what's now possible
Opinions of activists: A few thousand interviews billions of
political opinions in social media posts (1B every 2 Days)
Exercise: A survey: How many times did you exercise last
week? 500K people carrying cell phones with
accelerometers
Social contacts:
3/10

23. Examples of what's now possible
Opinions of activists: A few thousand interviews billions of
political opinions in social media posts (1B every 2 Days)
Exercise: A survey: How many times did you exercise last
week? 500K people carrying cell phones with
accelerometers
Social contacts: A survey: Please tell me your 5 best
friends
3/10

24. Examples of what's now possible
Opinions of activists: A few thousand interviews billions of
political opinions in social media posts (1B every 2 Days)
Exercise: A survey: How many times did you exercise last
week? 500K people carrying cell phones with
accelerometers
Social contacts: A survey: Please tell me your 5 best
friends continuous record of phone calls, emails, text
messages, bluetooth, social media connections, address books
3/10

25. Examples of what's now possible
Opinions of activists: A few thousand interviews billions of
political opinions in social media posts (1B every 2 Days)
Exercise: A survey: How many times did you exercise last
week? 500K people carrying cell phones with
accelerometers
Social contacts: A survey: Please tell me your 5 best
friends continuous record of phone calls, emails, text
messages, bluetooth, social media connections, address books
Economic development in developing countries:
3/10

26. Examples of what's now possible
Opinions of activists: A few thousand interviews billions of
political opinions in social media posts (1B every 2 Days)
Exercise: A survey: How many times did you exercise last
week? 500K people carrying cell phones with
accelerometers
Social contacts: A survey: Please tell me your 5 best
friends continuous record of phone calls, emails, text
messages, bluetooth, social media connections, address books
Economic development in developing countries: Dubious or
nonexistent governmental statistics
3/10

27. Examples of what's now possible
Opinions of activists: A few thousand interviews billions of
political opinions in social media posts (1B every 2 Days)
Exercise: A survey: How many times did you exercise last
week? 500K people carrying cell phones with
accelerometers
Social contacts: A survey: Please tell me your 5 best
friends continuous record of phone calls, emails, text
messages, bluetooth, social media connections, address books
Economic development in developing countries: Dubious or
nonexistent governmental statistics satellite images of
human-generated light at night, road networks, other
infrastructure
3/10

28. Examples of what's now possible
Opinions of activists: A few thousand interviews billions of
political opinions in social media posts (1B every 2 Days)
Exercise: A survey: How many times did you exercise last
week? 500K people carrying cell phones with
accelerometers
Social contacts: A survey: Please tell me your 5 best
friends continuous record of phone calls, emails, text
messages, bluetooth, social media connections, address books
Economic development in developing countries: Dubious or
nonexistent governmental statistics satellite images of
human-generated light at night, road networks, other
infrastructure
Many, many, more. . .
3/10

29. Examples of what's now possible
Opinions of activists: A few thousand interviews billions of
political opinions in social media posts (1B every 2 Days)
Exercise: A survey: How many times did you exercise last
week? 500K people carrying cell phones with
accelerometers
Social contacts: A survey: Please tell me your 5 best
friends continuous record of phone calls, emails, text
messages, bluetooth, social media connections, address books
Economic development in developing countries: Dubious or
nonexistent governmental statistics satellite images of
human-generated light at night, road networks, other
infrastructure
Many, many, more. . .
In each: without new analytics, the data are useless
3/10

30. The End of The Quantitative-Qualitative Divide
4/10

31. The End of The Quantitative-Qualitative Divide
The Quant-Qual divide exists in every

32. eld.
4/10

33. The End of The Quantitative-Qualitative Divide
The Quant-Qual divide exists in every

34. eld.
Qualitative researchers: overwhelmed by information; need
help
4/10

35. The End of The Quantitative-Qualitative Divide
The Quant-Qual divide exists in every

36. eld.
Qualitative researchers: overwhelmed by information; need
help
Quantitative researchers: recognize the huge amounts of
information in qualitative analyses, starting to analyze
unstructured text, video, audio as data
4/10

37. The End of The Quantitative-Qualitative Divide
The Quant-Qual divide exists in every

38. eld.
Qualitative researchers: overwhelmed by information; need
help
Quantitative researchers: recognize the huge amounts of
information in qualitative analyses, starting to analyze
unstructured text, video, audio as data
Expert-vs-analytics contests: Whenever enough information is
quanti

39. ed, a right answer exists, and good analytics are
applied: analytics wins
4/10

40. The End of The Quantitative-Qualitative Divide
The Quant-Qual divide exists in every

41. eld.
Qualitative researchers: overwhelmed by information; need
help
Quantitative researchers: recognize the huge amounts of
information in qualitative analyses, starting to analyze
unstructured text, video, audio as data
Expert-vs-analytics contests: Whenever enough information is
quanti

42. ed, a right answer exists, and good analytics are
applied: analytics wins
Moral of the story:
4/10

43. The End of The Quantitative-Qualitative Divide
The Quant-Qual divide exists in every

44. eld.
Qualitative researchers: overwhelmed by information; need
help
Quantitative researchers: recognize the huge amounts of
information in qualitative analyses, starting to analyze
unstructured text, video, audio as data
Expert-vs-analytics contests: Whenever enough information is
quanti

45. ed, a right answer exists, and good analytics are
applied: analytics wins
Moral of the story:
Fully human is inadequate
4/10

46. The End of The Quantitative-Qualitative Divide
The Quant-Qual divide exists in every

47. eld.
Qualitative researchers: overwhelmed by information; need
help
Quantitative researchers: recognize the huge amounts of
information in qualitative analyses, starting to analyze
unstructured text, video, audio as data
Expert-vs-analytics contests: Whenever enough information is
quanti

48. ed, a right answer exists, and good analytics are
applied: analytics wins
Moral of the story:
Fully human is inadequate
Fully automated fails
4/10

49. The End of The Quantitative-Qualitative Divide
The Quant-Qual divide exists in every

50. eld.
Qualitative researchers: overwhelmed by information; need
help
Quantitative researchers: recognize the huge amounts of
information in qualitative analyses, starting to analyze
unstructured text, video, audio as data
Expert-vs-analytics contests: Whenever enough information is
quanti

51. ed, a right answer exists, and good analytics are
applied: analytics wins
Moral of the story:
Fully human is inadequate
Fully automated fails
We need computer assisted, human controlled technology
4/10

52. The End of The Quantitative-Qualitative Divide
The Quant-Qual divide exists in every

53. eld.
Qualitative researchers: overwhelmed by information; need
help
Quantitative researchers: recognize the huge amounts of
information in qualitative analyses, starting to analyze
unstructured text, video, audio as data
Expert-vs-analytics contests: Whenever enough information is
quanti

54. ed, a right answer exists, and good analytics are
applied: analytics wins
Moral of the story:
Fully human is inadequate
Fully automated fails
We need computer assisted, human controlled technology
(Technically correct, politically much easier)
4/10

55. How to Read a Billion Blog Posts
Classify Deaths without Physicians
5/10

56. How to Read a Billion Blog Posts
Classify Deaths without Physicians
Examples of Bad Analytics:
5/10

57. How to Read a Billion Blog Posts
Classify Deaths without Physicians
Examples of Bad Analytics:
Physicians' Verbal Autopsy analysis
5/10

58. How to Read a Billion Blog Posts
Classify Deaths without Physicians
Examples of Bad Analytics:
Physicians' Verbal Autopsy analysis
Sentiment analysis via word counts
5/10

59. How to Read a Billion Blog Posts
Classify Deaths without Physicians
Examples of Bad Analytics:
Physicians' Verbal Autopsy analysis
Sentiment analysis via word counts
Dierent problems, Same Analytics Solution:
5/10

60. How to Read a Billion Blog Posts
Classify Deaths without Physicians
Examples of Bad Analytics:
Physicians' Verbal Autopsy analysis
Sentiment analysis via word counts
Dierent problems, Same Analytics Solution:
Key to both methods: classifying (deaths, social media posts)
5/10

61. How to Read a Billion Blog Posts
Classify Deaths without Physicians
Examples of Bad Analytics:
Physicians' Verbal Autopsy analysis
Sentiment analysis via word counts
Dierent problems, Same Analytics Solution:
Key to both methods: classifying (deaths, social media posts)
Key to both goals: estimating %'s
5/10

62. How to Read a Billion Blog Posts
Classify Deaths without Physicians
Examples of Bad Analytics:
Physicians' Verbal Autopsy analysis
Sentiment analysis via word counts
Dierent problems, Same Analytics Solution:
Key to both methods: classifying (deaths, social media posts)
Key to both goals: estimating %'s
Modern Data Analytics: New method led to:
5/10

63. How to Read a Billion Blog Posts
Classify Deaths without Physicians
Examples of Bad Analytics:
Physicians' Verbal Autopsy analysis
Sentiment analysis via word counts
Dierent problems, Same Analytics Solution:
Key to both methods: classifying (deaths, social media posts)
Key to both goals: estimating %'s
Modern Data Analytics: New method led to:
1.
5/10

64. How to Read a Billion Blog Posts
Classify Deaths without Physicians
Examples of Bad Analytics:
Physicians' Verbal Autopsy analysis
Sentiment analysis via word counts
Dierent problems, Same Analytics Solution:
Key to both methods: classifying (deaths, social media posts)
Key to both goals: estimating %'s
Modern Data Analytics: New method led to:
1.
2. Worldwide cause-of-death estimates for
5/10

65. The Solvency of Social Security
6/10

66. The Solvency of Social Security
Successful: single largest government program; lifted a whole
generation out of poverty; extremely popular
6/10

67. The Solvency of Social Security
Successful: single largest government program; lifted a whole
generation out of poverty; extremely popular
Solvency: depends on mortality forecasts:
6/10

68. The Solvency of Social Security
Successful: single largest government program; lifted a whole
generation out of poverty; extremely popular
Solvency: depends on mortality forecasts: If retirees receive
bene

69. ts longer than expected, the Trust Fund runs out
6/10

70. The Solvency of Social Security
Successful: single largest government program; lifted a whole
generation out of poverty; extremely popular
Solvency: depends on mortality forecasts: If retirees receive
bene

71. ts longer than expected, the Trust Fund runs out
SSA data: little change other than updates for 75 years
6/10

72. The Solvency of Social Security
Successful: single largest government program; lifted a whole
generation out of poverty; extremely popular
Solvency: depends on mortality forecasts: If retirees receive
bene

73. ts longer than expected, the Trust Fund runs out
SSA data: little change other than updates for 75 years
SSA analytics:
6/10

74. The Solvency of Social Security
Successful: single largest government program; lifted a whole
generation out of poverty; extremely popular
Solvency: depends on mortality forecasts: If retirees receive
bene

75. ts longer than expected, the Trust Fund runs out
SSA data: little change other than updates for 75 years
SSA analytics:
Few statistical improvements for 75 years
6/10

76. The Solvency of Social Security
Successful: single largest government program; lifted a whole
generation out of poverty; extremely popular
Solvency: depends on mortality forecasts: If retirees receive
bene

77. ts longer than expected, the Trust Fund runs out
SSA data: little change other than updates for 75 years
SSA analytics:
Few statistical improvements for 75 years
Ignore risk factors (smoking, obesity)
6/10

78. The Solvency of Social Security
Successful: single largest government program; lifted a whole
generation out of poverty; extremely popular
Solvency: depends on mortality forecasts: If retirees receive
bene

79. ts longer than expected, the Trust Fund runs out
SSA data: little change other than updates for 75 years
SSA analytics:
Few statistical improvements for 75 years
Ignore risk factors (smoking, obesity)
Mostly informal (subject to error political in
uence)
6/10

80. The Solvency of Social Security
Successful: single largest government program; lifted a whole
generation out of poverty; extremely popular
Solvency: depends on mortality forecasts: If retirees receive
bene

81. ts longer than expected, the Trust Fund runs out
SSA data: little change other than updates for 75 years
SSA analytics:
Few statistical improvements for 75 years
Ignore risk factors (smoking, obesity)
Mostly informal (subject to error political in
uence)
Forecasts: inaccurate, inconsistent, overly optimistic
6/10

82. The Solvency of Social Security
Successful: single largest government program; lifted a whole
generation out of poverty; extremely popular
Solvency: depends on mortality forecasts: If retirees receive
bene

83. ts longer than expected, the Trust Fund runs out
SSA data: little change other than updates for 75 years
SSA analytics:
Few statistical improvements for 75 years
Ignore risk factors (smoking, obesity)
Mostly informal (subject to error political in
uence)
Forecasts: inaccurate, inconsistent, overly optimistic
New customized analytics we developed:
6/10

84. The Solvency of Social Security
Successful: single largest government program; lifted a whole
generation out of poverty; extremely popular
Solvency: depends on mortality forecasts: If retirees receive
bene

85. ts longer than expected, the Trust Fund runs out
SSA data: little change other than updates for 75 years
SSA analytics:
Few statistical improvements for 75 years
Ignore risk factors (smoking, obesity)
Mostly informal (subject to error political in
uence)
Forecasts: inaccurate, inconsistent, overly optimistic
New customized analytics we developed:
Logical consistency (e.g., older people have higher mortality)
6/10

86. The Solvency of Social Security
Successful: single largest government program; lifted a whole
generation out of poverty; extremely popular
Solvency: depends on mortality forecasts: If retirees receive
bene

87. ts longer than expected, the Trust Fund runs out
SSA data: little change other than updates for 75 years
SSA analytics:
Few statistical improvements for 75 years
Ignore risk factors (smoking, obesity)
Mostly informal (subject to error political in
uence)
Forecasts: inaccurate, inconsistent, overly optimistic
New customized analytics we developed:
Logical consistency (e.g., older people have higher mortality)
More accurate forecasts
6/10

88. The Solvency of Social Security
Successful: single largest government program; lifted a whole
generation out of poverty; extremely popular
Solvency: depends on mortality forecasts: If retirees receive
bene

89. ts longer than expected, the Trust Fund runs out
SSA data: little change other than updates for 75 years
SSA analytics:
Few statistical improvements for 75 years
Ignore risk factors (smoking, obesity)
Mostly informal (subject to error political in
uence)
Forecasts: inaccurate, inconsistent, overly optimistic
New customized analytics we developed:
Logical consistency (e.g., older people have higher mortality)
More accurate forecasts
Trust fund needs $800 billion more than SSA thought
6/10

90. The Solvency of Social Security
Successful: single largest government program; lifted a whole
generation out of poverty; extremely popular
Solvency: depends on mortality forecasts: If retirees receive
bene

91. ts longer than expected, the Trust Fund runs out
SSA data: little change other than updates for 75 years
SSA analytics:
Few statistical improvements for 75 years
Ignore risk factors (smoking, obesity)
Mostly informal (subject to error political in
uence)
Forecasts: inaccurate, inconsistent, overly optimistic
New customized analytics we developed:
Logical consistency (e.g., older people have higher mortality)
More accurate forecasts
Trust fund needs $800 billion more than SSA thought
Other applications to insurance industry, public health, etc.
6/10

92. Following Conversations that Hide in Plain Sight
7/10

93. Following Conversations that Hide in Plain Sight
Example Substitution 1:
7/10

94. Following Conversations that Hide in Plain Sight
Example Substitution 1:
ê1
7/10

95. Following Conversations that Hide in Plain Sight
Example Substitution 1:
ê1 Freedom
7/10

96. Following Conversations that Hide in Plain Sight
Example Substitution 1:
ê1 Freedom
7/10

97. Following Conversations that Hide in Plain Sight
Example Substitution 1:
ê1 Freedom
î0
7/10

98. Following Conversations that Hide in Plain Sight
Example Substitution 1:
ê1 Freedom
î0 Eye

99. eld
7/10

100. Following Conversations that Hide in Plain Sight
Example Substitution 1:
ê1 Freedom
î0 Eye

101. eld (nonsensical)
7/10

102. Following Conversations that Hide in Plain Sight
Example Substitution 1: Homograph
ê1 Freedom
î0 Eye

103. eld (nonsensical)
7/10

104. Following Conversations that Hide in Plain Sight
Example Substitution 1: Homograph
ê1 Freedom
î0 Eye

105. eld (nonsensical)
7/10

106. Following Conversations that Hide in Plain Sight
Example Substitution 1: Homograph
ê1 Freedom
î0 Eye

107. eld (nonsensical)
Example Substitution 2:
7/10

108. Following Conversations that Hide in Plain Sight
Example Substitution 1: Homograph
ê1 Freedom
î0 Eye

109. eld (nonsensical)
Example Substitution 2:
Œ
7/10

110. Following Conversations that Hide in Plain Sight
Example Substitution 1: Homograph
ê1 Freedom
î0 Eye

111. eld (nonsensical)
Example Substitution 2:
ΠHarmonious [Society] (ocial slogan)
7/10

112. Following Conversations that Hide in Plain Sight
Example Substitution 1: Homograph
ê1 Freedom
î0 Eye

113. eld (nonsensical)
Example Substitution 2:
ΠHarmonious [Society] (ocial slogan)
7/10

114. Following Conversations that Hide in Plain Sight
Example Substitution 1: Homograph
ê1 Freedom
î0 Eye

115. eld (nonsensical)
Example Substitution 2:
ΠHarmonious [Society] (ocial slogan)
³ù
7/10

116. Following Conversations that Hide in Plain Sight
Example Substitution 1: Homograph
ê1 Freedom
î0 Eye

117. eld (nonsensical)
Example Substitution 2:
ΠHarmonious [Society] (ocial slogan)
³ù River crab
7/10

118. Following Conversations that Hide in Plain Sight
Example Substitution 1: Homograph
ê1 Freedom
î0 Eye

119. eld (nonsensical)
Example Substitution 2:
ΠHarmonious [Society] (ocial slogan)
³ù River crab (irrelevant)
7/10

120. Following Conversations that Hide in Plain Sight
Example Substitution 1: Homograph
ê1 Freedom
î0 Eye

121. eld (nonsensical)
Example Substitution 2: Homophone (sound like hexie)
ΠHarmonious [Society] (ocial slogan)
³ù River crab (irrelevant)
7/10

122. Following Conversations that Hide in Plain Sight
Example Substitution 1: Homograph
ê1 Freedom
î0 Eye

123. eld (nonsensical)
Example Substitution 2: Homophone (sound like hexie)
ΠHarmonious [Society] (ocial slogan)
³ù River crab (irrelevant)
They can't follow the conversation; Our methods can!
7/10

124. Following Conversations that Hide in Plain Sight
Example Substitution 1: Homograph
ê1 Freedom
î0 Eye

125. eld (nonsensical)
Example Substitution 2: Homophone (sound like hexie)
ΠHarmonious [Society] (ocial slogan)
³ù River crab (irrelevant)
They can't follow the conversation; Our methods can!
The same task:
7/10

126. Following Conversations that Hide in Plain Sight
Example Substitution 1: Homograph
ê1 Freedom
î0 Eye

127. eld (nonsensical)
Example Substitution 2: Homophone (sound like hexie)
ΠHarmonious [Society] (ocial slogan)
³ù River crab (irrelevant)
They can't follow the conversation; Our methods can!
The same task: (1) Government and industry analyst's job,
7/10

128. Following Conversations that Hide in Plain Sight
Example Substitution 1: Homograph
ê1 Freedom
î0 Eye

129. eld (nonsensical)
Example Substitution 2: Homophone (sound like hexie)
ΠHarmonious [Society] (ocial slogan)
³ù River crab (irrelevant)
They can't follow the conversation; Our methods can!
The same task: (1) Government and industry analyst's job, (2)
language drift (#BostonBombings #BostonStrong),
7/10

130. Following Conversations that Hide in Plain Sight
Example Substitution 1: Homograph
ê1 Freedom
î0 Eye

131. eld (nonsensical)
Example Substitution 2: Homophone (sound like hexie)
ΠHarmonious [Society] (ocial slogan)
³ù River crab (irrelevant)
They can't follow the conversation; Our methods can!
The same task: (1) Government and industry analyst's job, (2)
language drift (#BostonBombings #BostonStrong), (3) Child
pornographers,
7/10

132. Following Conversations that Hide in Plain Sight
Example Substitution 1: Homograph
ê1 Freedom
î0 Eye

133. eld (nonsensical)
Example Substitution 2: Homophone (sound like hexie)
ΠHarmonious [Society] (ocial slogan)
³ù River crab (irrelevant)
They can't follow the conversation; Our methods can!
The same task: (1) Government and industry analyst's job, (2)
language drift (#BostonBombings #BostonStrong), (3) Child
pornographers, (4) Look-alike modeling,
7/10

134. Following Conversations that Hide in Plain Sight
Example Substitution 1: Homograph
ê1 Freedom
î0 Eye

135. eld (nonsensical)
Example Substitution 2: Homophone (sound like hexie)
ΠHarmonious [Society] (ocial slogan)
³ù River crab (irrelevant)
They can't follow the conversation; Our methods can!
The same task: (1) Government and industry analyst's job, (2)
language drift (#BostonBombings #BostonStrong), (3) Child
pornographers, (4) Look-alike modeling,(5) Starting point for
sophisticated automated text analysis
7/10

136. Computer-Assisted Reading (Consilience)
8/10

137. Computer-Assisted Reading (Consilience)
To understand many documents, humans create categories to
represent conceptualization, insight, etc.
8/10

138. Computer-Assisted Reading (Consilience)
To understand many documents, humans create categories to
represent conceptualization, insight, etc.
Most

139. rms: impose

140. xed categorizations to tally customer
complaints, sort reports, retrieve information
8/10

141. Computer-Assisted Reading (Consilience)
To understand many documents, humans create categories to
represent conceptualization, insight, etc.
Most

142. rms: impose

143. xed categorizations to tally customer
complaints, sort reports, retrieve information
Bad Analytics:
8/10

144. Computer-Assisted Reading (Consilience)
To understand many documents, humans create categories to
represent conceptualization, insight, etc.
Most

145. rms: impose

146. xed categorizations to tally customer
complaints, sort reports, retrieve information
Bad Analytics:
Unassisted Human Categorization: time consuming; huge
eorts trying not to innovate!
8/10

147. Computer-Assisted Reading (Consilience)
To understand many documents, humans create categories to
represent conceptualization, insight, etc.
Most

148. rms: impose

149. xed categorizations to tally customer
complaints, sort reports, retrieve information
Bad Analytics:
Unassisted Human Categorization: time consuming; huge
eorts trying not to innovate!
Fully Automated Cluster Analysis: Many widely available,
but none work (computers don't know what you want!)
8/10

150. Computer-Assisted Reading (Consilience)
To understand many documents, humans create categories to
represent conceptualization, insight, etc.
Most

151. rms: impose

152. xed categorizations to tally customer
complaints, sort reports, retrieve information
Bad Analytics:
Unassisted Human Categorization: time consuming; huge
eorts trying not to innovate!
Fully Automated Cluster Analysis: Many widely available,
but none work (computers don't know what you want!)
Our alternative: Computer-assisted Categorization
8/10

153. Computer-Assisted Reading (Consilience)
To understand many documents, humans create categories to
represent conceptualization, insight, etc.
Most

154. rms: impose

155. xed categorizations to tally customer
complaints, sort reports, retrieve information
Bad Analytics:
Unassisted Human Categorization: time consuming; huge
eorts trying not to innovate!
Fully Automated Cluster Analysis: Many widely available,
but none work (computers don't know what you want!)
Our alternative: Computer-assisted Categorization
You decide what's important, but with help
8/10

156. Computer-Assisted Reading (Consilience)
To understand many documents, humans create categories to
represent conceptualization, insight, etc.
Most

157. rms: impose

158. xed categorizations to tally customer
complaints, sort reports, retrieve information
Bad Analytics:
Unassisted Human Categorization: time consuming; huge
eorts trying not to innovate!
Fully Automated Cluster Analysis: Many widely available,
but none work (computers don't know what you want!)
Our alternative: Computer-assisted Categorization
You decide what's important, but with help
Invert eort: you innovate; the computer categorizes
8/10

159. Computer-Assisted Reading (Consilience)
To understand many documents, humans create categories to
represent conceptualization, insight, etc.
Most

160. rms: impose

161. xed categorizations to tally customer
complaints, sort reports, retrieve information
Bad Analytics:
Unassisted Human Categorization: time consuming; huge
eorts trying not to innovate!
Fully Automated Cluster Analysis: Many widely available,
but none work (computers don't know what you want!)
Our alternative: Computer-assisted Categorization
You decide what's important, but with help
Invert eort: you innovate; the computer categorizes
Insights: easier, faster, better
8/10

162. Computer-Assisted Reading (Consilience)
To understand many documents, humans create categories to
represent conceptualization, insight, etc.
Most

163. rms: impose

164. xed categorizations to tally customer
complaints, sort reports, retrieve information
Bad Analytics:
Unassisted Human Categorization: time consuming; huge
eorts trying not to innovate!
Fully Automated Cluster Analysis: Many widely available,
but none work (computers don't know what you want!)
Our alternative: Computer-assisted Categorization
You decide what's important, but with help
Invert eort: you innovate; the computer categorizes
Insights: easier, faster, better
(Lots of technology, but it's behind the scenes)
8/10

165. Example Insights from Computer-Assisted Reading
9/10

166. Example Insights from Computer-Assisted Reading
1. What Members of Congress Do
9/10

167. Example Insights from Computer-Assisted Reading
1. What Members of Congress Do
Data: 64,000 Senators' press releases
9/10

168. Example Insights from Computer-Assisted Reading
1. What Members of Congress Do
Data: 64,000 Senators' press releases
Categorization: (1) advertising, (2) position taking, (3) credit
claiming
9/10

169. Example Insights from Computer-Assisted Reading
1. What Members of Congress Do
Data: 64,000 Senators' press releases
Categorization: (1) advertising, (2) position taking, (3) credit
claiming
New Insight: partisan taunting
9/10

170. Example Insights from Computer-Assisted Reading
1. What Members of Congress Do
Data: 64,000 Senators' press releases
Categorization: (1) advertising, (2) position taking, (3) credit
claiming
New Insight: partisan taunting
Joe Wilson during Obama's State of the Union: You lie!
9/10

171. Example Insights from Computer-Assisted Reading
1. What Members of Congress Do
Data: 64,000 Senators' press releases
Categorization: (1) advertising, (2) position taking, (3) credit
claiming
New Insight: partisan taunting
Joe Wilson during Obama's State of the Union: You lie!
Senator Lautenberg Blasts Republicans as `Chicken Hawks'
9/10

172. Example Insights from Computer-Assisted Reading
1. What Members of Congress Do
Data: 64,000 Senators' press releases
Categorization: (1) advertising, (2) position taking, (3) credit
claiming
New Insight: partisan taunting
Joe Wilson during Obama's State of the Union: You lie!
Senator Lautenberg Blasts Republicans as `Chicken Hawks'
How common is it?
9/10

173. Example Insights from Computer-Assisted Reading
1. What Members of Congress Do
Data: 64,000 Senators' press releases
Categorization: (1) advertising, (2) position taking, (3) credit
claiming
New Insight: partisan taunting
Joe Wilson during Obama's State of the Union: You lie!
Senator Lautenberg Blasts Republicans as `Chicken Hawks'
How common is it? 27% of all Senatorial press releases!
9/10

174. Example Insights from Computer-Assisted Reading
1. What Members of Congress Do
Data: 64,000 Senators' press releases
Categorization: (1) advertising, (2) position taking, (3) credit
claiming
New Insight: partisan taunting
Joe Wilson during Obama's State of the Union: You lie!
Senator Lautenberg Blasts Republicans as `Chicken Hawks'
How common is it? 27% of all Senatorial press releases!
2. Reverse Engineering Chinese Censorship
9/10

175. Example Insights from Computer-Assisted Reading
1. What Members of Congress Do
Data: 64,000 Senators' press releases
Categorization: (1) advertising, (2) position taking, (3) credit
claiming
New Insight: partisan taunting
Joe Wilson during Obama's State of the Union: You lie!
Senator Lautenberg Blasts Republicans as `Chicken Hawks'
How common is it? 27% of all Senatorial press releases!
2. Reverse Engineering Chinese Censorship
Previous approach: manual eort to see what is taken down
9/10

176. Example Insights from Computer-Assisted Reading
1. What Members of Congress Do
Data: 64,000 Senators' press releases
Categorization: (1) advertising, (2) position taking, (3) credit
claiming
New Insight: partisan taunting
Joe Wilson during Obama's State of the Union: You lie!
Senator Lautenberg Blasts Republicans as `Chicken Hawks'
How common is it? 27% of all Senatorial press releases!
2. Reverse Engineering Chinese Censorship
Previous approach: manual eort to see what is taken down
Data: We get posts before the Chinese censor them
9/10

177. Example Insights from Computer-Assisted Reading
1. What Members of Congress Do
Data: 64,000 Senators' press releases
Categorization: (1) advertising, (2) position taking, (3) credit
claiming
New Insight: partisan taunting
Joe Wilson during Obama's State of the Union: You lie!
Senator Lautenberg Blasts Republicans as `Chicken Hawks'
How common is it? 27% of all Senatorial press releases!
2. Reverse Engineering Chinese Censorship
Previous approach: manual eort to see what is taken down
Data: We get posts before the Chinese censor them
We analyzed 11 million posts, about 13% censored
9/10

178. Example Insights from Computer-Assisted Reading
1. What Members of Congress Do
Data: 64,000 Senators' press releases
Categorization: (1) advertising, (2) position taking, (3) credit
claiming
New Insight: partisan taunting
Joe Wilson during Obama's State of the Union: You lie!
Senator Lautenberg Blasts Republicans as `Chicken Hawks'
How common is it? 27% of all Senatorial press releases!
2. Reverse Engineering Chinese Censorship
Previous approach: manual eort to see what is taken down
Data: We get posts before the Chinese censor them
We analyzed 11 million posts, about 13% censored
Previous understanding: they censor criticisms of the
government
9/10

179. Example Insights from Computer-Assisted Reading
1. What Members of Congress Do
Data: 64,000 Senators' press releases
Categorization: (1) advertising, (2) position taking, (3) credit
claiming
New Insight: partisan taunting
Joe Wilson during Obama's State of the Union: You lie!
Senator Lautenberg Blasts Republicans as `Chicken Hawks'
How common is it? 27% of all Senatorial press releases!
2. Reverse Engineering Chinese Censorship
Previous approach: manual eort to see what is taken down
Data: We get posts before the Chinese censor them
We analyzed 11 million posts, about 13% censored
Previous understanding: they censor criticisms of the
government
Results:
9/10

180. Example Insights from Computer-Assisted Reading
1. What Members of Congress Do
Data: 64,000 Senators' press releases
Categorization: (1) advertising, (2) position taking, (3) credit
claiming
New Insight: partisan taunting
Joe Wilson during Obama's State of the Union: You lie!
Senator Lautenberg Blasts Republicans as `Chicken Hawks'
How common is it? 27% of all Senatorial press releases!
2. Reverse Engineering Chinese Censorship
Previous approach: manual eort to see what is taken down
Data: We get posts before the Chinese censor them
We analyzed 11 million posts, about 13% censored
Previous understanding: they censor criticisms of the
government
Results:
Uncensored: criticism of the government
9/10

181. Example Insights from Computer-Assisted Reading
1. What Members of Congress Do
Data: 64,000 Senators' press releases
Categorization: (1) advertising, (2) position taking, (3) credit
claiming
New Insight: partisan taunting
Joe Wilson during Obama's State of the Union: You lie!
Senator Lautenberg Blasts Republicans as `Chicken Hawks'
How common is it? 27% of all Senatorial press releases!
2. Reverse Engineering Chinese Censorship
Previous approach: manual eort to see what is taken down
Data: We get posts before the Chinese censor them
We analyzed 11 million posts, about 13% censored
Previous understanding: they censor criticisms of the
government
Results:
Uncensored: criticism of the government
Censored: attempts at collective action
9/10

182. For more information
GaryKing.org
10/10