Adventure in Data: A tour of visualization projects at Twitter

1. Krist Wongsuphasawat / @kristw Adventure in data A tour of visualizations at Twitter

2. Krist Wongsuphasawat / @kristw

3. Computer Engineer Bangkok, Thailand PhD in Computer Science Univ. of Maryland Information Visualization IBM Microsoft Data Visualization Scientist Twitter Krist Wongsuphasawat / @kristw

4. Krist Wongsuphasawat / @kristw Adventure in data A whirlwind tour of visualization projects at Twitter

5. Get data

6. Having all Tweets How people think I feel.

7. How people think I feel. How I really feel. Having all Tweets

8. • Too much data. Want only relevant Tweets • hashtag: #BRA • keywords: “goal” • Need to aggregate & reduce size • Long processing time (hours) Challenges

9. Hadoop Cluster Data Storage Workﬂow

10. Hadoop Cluster Pig / Scalding (slow) Data Storage Tool Workﬂow

11. Hadoop Cluster Pig / Scalding (slow) Data Storage Tool Workﬂow

12. Hadoop Cluster Pig / Scalding (slow) Data Storage Tool Smaller datasetYour laptop Workﬂow

13. Hadoop Cluster Pig / Scalding (slow) Data Storage Tool Final dataset Tool node.js / python / excel (fast) Your laptop Workﬂow Smaller dataset

14. Krist Wongsuphasawat / @kristw Cleaning data a story of my life

15. Storytelling Analytics Tools Creative Projects

16. To understand the world and share the stories To understand Twitter users and improve the service To showcase the data and inspire Projects Storytelling Analytics Tools Creative

17. Storytelling1 World Cup Election Oscars TV Shows New Year Earthquake Super Bowl Protest … Behaviors Sleeping Daylight saving Language … Events Fasting Information spread Commute

18. So many things we could learn from Twitter data

19. Give us interesting vis about xxxx by Nov 10

20. Challenge accepted

21. Tweets (+ media) photos, videos What? Where? When? GEO TIME TEXT

22. What? Where? When? GEO TIME TEXT Visualize Tweets

24. Time Tweets/second

25. Time Tweets/second

26. Time Tweets/second + Annotation http://www.ﬂickr.com/photos/twitterofﬁce/5681263084/

28. Geo Heatmap Low density High density

29. Geo San Francisco ﬂickr.com/photos/twitterofﬁce/8798020541 Low density High density

30. Geo San Francisco Rebuild the world based on tweet volumes twitter.github.io/interactive/andes/

32. Text www.wordle.net Some experiments during World Cup

33. Text www.wordle.net Word cloud of Tweets right after the 1st goal

34. Text Word cloud of Tweets right after the 1st goal www.wordle.net It was an “own” goal.

35. Text WordTree [Wattenberg & Viégas 2008] www.jasondavies.com/wordtre www.jasondavies.com/wordtree

36. Text word/phrase/hashtag count topic

38. Time + Geo Tweet pattern [Rios & Lin 2012] Night Late night Daytime Night Late night Daytime

39. Night Late night Daytime Night Late night Daytime Time + Geo Tweet pattern [Rios & Lin 2012]

43. Geo + Text Real-time Tweet map

44. Geo + Text Real-time Tweet map

45. most frequent term Geo + Text Real-time Tweet map

46. Gmail was down Jan 24, 2014 Geo + Text Real-time Tweet map

47. Nelson Mandela passed away Dec 5, 2013 Geo + Text Real-time Tweet map

49. Time + Text UEFA Champions League Biggest tournament for European soccer clubs Many Tweets during the matches

50. UEFA Champions League Dortmund Bayern Munich Count Tweets mentioning the teams every minute Team 1 Team 2 Time + Text

51. Time + Text UEFA Champions League

52. + “goal” count + context Time + Text UEFA Champions League

53. + “offside” Time + Text UEFA Champions League

54. + players Time + Text UEFA Champions League

55. A B C D A C C Competition Tree vs vs vs + = uclﬁnal.twitter.com

57. Time + Text + Geo State of the Union twitter.github.io/interactive/sotu2014

58. 1) timeline + topic from Tweets 4) Density map of Tweets about selected topic 3) Volume of Tweets by topics during selected part of the SOTU 2) context (speech) twitter.github.io/interactive/sotu2014 Time + Text + Geo State of the Union

59. World Cup 2014Time + Text interactive.twitter.com/wccompetitree

60. Time + Text + Geo World Cup 2014 interactive.twitter.com/wccompetitree

62. What? Where? When? GEO TIME TEXT Visualize Tweets + Non-Twitter data CONTEXT

63. Time + Text New Year 2014

64. Time + Text New Year 2014

65. Time + Text + Geo (c) New Year 2014 twitter.github.io/interactive/newyear2014/

66. Analytics Tools2 Data sources Output explore analyze present get * *

67. Analytics Tools2 Data sources Output explore analyze present get * *

68. Analytics Tools2 Data sources Output explore analyze present get * * ad-hoc scripts

69. Analytics Tools2 Data sources Output explore analyze present get * * ad-hoc scripts tools for exploration

70. User activity logs

71. UsersUseTwitter

72. UsersUse Product Managers Curious Twitter

73. UsersUse Curious Engineers Log data in Hadoop Write Twitter Instrument Product Managers

74. What are being logged? tweet activities

75. What are being logged? tweet from home timeline on twitter.com tweet from search page on iPhone activities

76. What are being logged? tweet from home timeline on twitter.com tweet from search page on iPhone sign up log in retweet etc. activities

77. Organize?

78. log event a.k.a. “client event” [Lee et al. 2012]

79. log event a.k.a. “client event” client : page : section : component : element : action web : home : timeline : tweet_box : button : tweet 1) User ID 2) Timestamp 3) Event name 4) Event detail [Lee et al. 2012]

80. Log data

81. UsersUse Curious Engineers Log data in Hadoop Twitter Instrument Write Product Managers bigger than Tweet data

82. UsersUse Curious Engineers Log data in Hadoop Data Scientists Ask Twitter Instrument Write Product Managers

83. UsersUse Curious Engineers Log data in Hadoop Data Scientists Find Ask Twitter Instrument Write Product Managers

84. Log data

85. UsersUse Curious Engineers Log data in Hadoop Data Scientists Find, Clean Ask Twitter Instrument Write Product Managers

86. UsersUse Curious Engineers Log data in Hadoop Data Scientists Find, Clean Ask Monitor Twitter Instrument Write Product Managers

87. UsersUse Curious Engineers Log data in Hadoop Data Scientists Find, Clean, Analyze Ask Monitor Twitter Instrument Write Product Managers

88. Log data EngineersData Scientists Usersin Hadoop Find, Clean, Analyze Use Monitor Ask Curious 1 2 Twitter Instrument Write Product Managers

89. Part I Find & Monitor Client Events

90. Motivation

91. Log data in Hadoop Engineers & Data Scientists billions of rows

92. Log data in Hadoop Aggregate 10,000+ event types date client page section comp. elem. action count 20141011 web home home - - impression 100 20141011 web home wtf - - click 20 Engineers & Data Scientists Client event collection

93. Log data in Hadoop Aggregate 10,000+ event types date client page section comp. elem. action count 20141011 web home home - - impression 100 20141011 web home wtf - - click 20 Engineers & Data Scientists Client event collection (Who-to-Follow)

94. Log data in Hadoop Aggregate Client event collection Engineers & Data Scientists

95. Log data in Hadoop Aggregate Find client page section component element action Search Client event collection Engineers & Data Scientists

96. Log data in Hadoop Aggregate Find client page section component element action Search Client event collection Engineers & Data Scientists

97. section? component? element?

98. client page section component element action Search Find Log data in Hadoop Aggregate web home * * impression* Client event collection Engineers & Data Scientists

99. client page section component element action Search Find Query Return Log data in Hadoop Results web : home : home : - : - : impression web : home : wtf : - : - : impression Aggregate web home * * impression* Client event collection Engineers & Data Scientists

100. client page section component element action Search Find Query Return Log data in Hadoop Results web : home : home : - : - : impression web : home : wtf : - : - : impression Aggregate search can be better Client event collection Engineers & Data Scientists

101. client page section component element action Search Find Query Return Log data in Hadoop Results web : home : home : - : - : impression web : home : wtf : - : - : impression Aggregate 10,000+ event types search can be better Client event collection Engineers & Data Scientists

102. client page section component element action Search Find Query Return Log data in Hadoop Results web : home : home : - : - : impression web : home : wtf : - : - : impression Aggregate search can be better 10,000+ event types not everybody knows What are all sections under web:home? Client event collection Engineers & Data Scientists

103. client page section component element action Search Find Query Return Log data in Hadoop Results web : home : home : - : - : impression Aggregate search can be better one graph / event 10,000+ event types not everybody knows What are all sections under web:home? Client event collection Engineers & Data Scientists

104. client page section component element action Search Find Query Return Log data in Hadoop Results web : home : home : - : - : impression Aggregate search can be better one graph / event x 10,000 10,000+ event types not everybody knows What are all sections under web:home? Client event collection Engineers & Data Scientists

105. ! • Search for client events • Explore client event collection • Monitor changes Goals

106. Design

107. Client event collection Engineers & Data Scientists

108. See Client event collection Engineers & Data Scientists

109. See Interactions search box => ﬁlter Client event collection narrow down Engineers & Data Scientists

110. See How to visualize? narrow down Client event collection Engineers & Data Scientists Interactions search box => ﬁlter

111. See How to visualize? narrow down Client event collection Engineers & Data Scientists client : page : section : component : element : actionInteractions search box => ﬁlter

112. Client event hierarchy iphone home - - - impression tweet tweet click iphone:home:-:-:-:impression iphone:home:-:tweet:tweet:click

113. Detect changes iphone home - - - impression tweet tweet click iphone home - - - impression tweet tweet click TODAY 7 DAYS AGO compared to

114. Calculate changes +5% +5% +5% +10% +10% +10% -5% -5% -5% DIFF

115. Display changes iphone home - - - impression tweet tweet click Map of the Market [Wattenberg 1999], StemView [Guerra-Gomez et al. 2013]

116. Display changes home - - - impression tweet tweet click iphone

117. Demo Scribe Radar

118. Twitter for Banana

119. Part II Analysis

120. Count page visits banana : home : - : - : - : impression home page

121. Funnel home page proﬁle page

122. Funnel analysis banana : home : - : - : - : impression banana : proﬁle : - : - : - : impression 1 jobhome page proﬁle page 1 hour

123. Funnel analysis banana : home : - : - : - : impression banana : proﬁle : - : - : - : impression banana : search : - : - : - : impression home page proﬁle page search page 2 jobs 2 hours

124. Funnel analysis banana : home : - : - : - : impression banana : proﬁle : - : - : - : impression banana : search : - : - : - : impression home page proﬁle page search page Specify all funnels manually! n jobs n hours

125. Goal banana : home : - : - : - : impression … …… 1 job => all funnels, visualized home page

126. • Visualize an overview of event sequences ! • Big data? eBay checkout sequences Related work [Wongsuphasawat et al. 2011, Monroe et al. 2013, …] [Shen et al. 2013]

127. User sessions Session#1 A B start end Session#4 start end A Session#2 B start end A Session#3 C start end A

128. Aggregate 4 sessions A BB C start end endend A A end A

129. Aggregate A BB C start end endend end 4 sessions

130. Aggregate C start end endend end A B 4 sessions

131. Aggregate C start end endend end A B 4 sessions

132. Aggregate C start end endend A B end 4 sessions

133. Aggregate C start endend A B end 4 sessions

134. Aggregate C start endend A B end 4 sessions

135. Aggregate start endend A CB end 4 sessions

136. Aggregate 4,000,000 sessions endend A CB end start

137. Twitter for Banana

138. try with sample data (~millions sessions, 10,000+ event types) ! original paper (100,000 sessions, ~10 event types)

139. fail…

140. How to make it work?

141. # of unique sequences

142. 1. Reduce event types Reduce # of unique sequences

143. 1. Reduce event types Reduce # of unique sequences 10,000 types select tweet sign up log out

144. 1. Reduce event types Reduce # of unique sequences 10,000 types select tweet sign up log out

145. 1. Reduce event types Reduce # of unique sequences 10,000 types select merge tweet from home timeline tweet from search page tweet … = tweet

146. 1. Reduce event types 2. Reduce sequence length Reduce # of unique sequences

147. 1. Reduce event types 2. Reduce sequence length Reduce # of unique sequences session 1000 events

148. 1. Reduce event types 2. Reduce sequence length Reduce # of unique sequences session 10 events after (window size & direction) 1000 events visit home page (alignment)

149. 1. Reduce event types 2. Reduce sequence length Reduce # of unique sequences Ask users for input}

150. 1. Reduce event types 2. Reduce sequence length 3. More aggregation on Hadoop Reduce # of unique sequences Ask users for input}

151. Collapse events Sequence ABBBCCCC ABBCC ABC ABCCCC ABCD ABCCCD ABCCE ABCDF ABCDG ABCDH e.g. tweet, tweet, tweet, … = tweet

152. Sequence ABC ABC ABC ABC ABCD ABCD ABCE ABCDF ABCDG ABCDH Collapse events

153. Group & Count Sequence ABC ABCD ABCE ABCDF ABCDG ABCDH … Count 2000 80 20 1 1 1 …

154. Group & Count Sequence ABC ABCD ABCE ABCDF ABCDG ABCDH ABCDI ABCDJK ABCDJL Count 2000 80 20 1 1 1 1 1 1 rare sequences (count < threshold)

155. Truncate Sequence ABC ABCD ABCE ABCDx ABCDx ABCDx ABCDx ABCDJx ABCDJx Count 2000 80 20 1 1 1 1 1 1 Replace last event with x (…)

156. Sequence ABC ABCD ABCE ABCDx ABCDJx Count 2000 80 20 4 2 Group & Count

157. Truncate more Sequence ABC ABCD ABCE ABCDx ABCDx Count 2000 80 20 4 2

158. Group & Count Sequence ABC ABCD ABCE ABCDx Count 2000 80 20 6

159. 1. Deﬁne set of events 2. Pick alignment, direction and window size 3. Run Hadoop job (with more aggregation) 4. Wait for it… (2+ hrs) 5. Visualize Final process ~100,000 patterns (10MB) gazillion patterns (TBs)

160. Demo Flying Sessions

161. • Large-scale User Activity Logs + Visual Analytics • Used in day-to-day operations at Twitter • Generalize to smaller systems Summary Challenge big data small data visualize & interact aggregate & sacriﬁce

162. Data sources Output Creative3 …

163. https://medium.com/@kristw/designing-the-game-of-tweets-7f87c30dc5a2 Demo / Game of Tweets

164. To understand the world and share the stories To understand Twitter users and improve the service To showcase the data and inspire Projects Storytelling Analytics Tools Creative

165. Oh no…. NOT AGAIN

166. To understand the world and share the stories To understand Twitter users and improve the service To showcase the data and inspire Projects Storytelling Analytics Tools Creative Reusable Toolkits To implement once and for all

167. Coming soon Demo / Labella.js

168. https://github.com/twitter/d3kit Demo / d3Kit http://www.slideshare.net/kristw/d3kit

169. Conclusions Data are everywhere. Many applications:   Journalism, Product development, Art, etc. Combine visualization with other skills:   HCI, Design, Stats, ML, etc. Don’t repeat yourself. Krist Wongsuphasawat / @kristw interactive.twitter.com kristw.yellowpigz.com

170. @philogb @trebor @miguelrios @smrogers @lintool @linuslee @chuangl4 and many other colleagues at @twitter Acknowledgement

171. Conclusions Data are everywhere. Many applications:   Journalism, Product development, Art, etc. Combine visualization with other skills:   HCI, Design, Stats, ML, etc. Don’t repeat yourself. Krist Wongsuphasawat / @kristw interactive.twitter.com kristw.yellowpigz.com

172. Thank you

173. Questions?