The document outlines the process and expectations in data visualization, particularly in the context of working with large datasets and varied client requirements. It emphasizes the importance of data wrangling, the time-consuming nature of data cleaning, and the need for effective communication and documentation throughout the project lifecycle. It also highlights different types of projects and the necessity to adapt visualization strategies based on audience and purpose, while expecting unexpected challenges and revisions.

1. WHAT TO EXPECT
WHEN YOU ARE
VISUALIZING
Krist Wongsuphasawat / @kristw
Based on true stories
Forever querying
Never-ending cleaning
Hopelessly prototyping
Last minute coding
and many more…

2. Computer Engineer
Bangkok, Thailand
PhD in Computer Science
Information Visualization
Univ. of Maryland
IBM
Microsoft
Data Visualization Scientist
Twitter
Krist Wongsuphasawat / @kristw

4. VISUALIZE DATA

5. INPUT
(DATA)
=YOU+ OUTPUT
(VIS)

6. EXPECT THE MISMATCHES

7. INPUT (DATA)
What clients think they have

8. INPUT (DATA)
What clients think they have What they usually have

9. YOU
What clients think you are

10. YOU
What clients think you are What they will get

11. OUTPUT (VIS)
What clients ask for

12. OUTPUT (VIS)
What clients ask for What they really need

13. I need this. Take this.

14. I need this. Here you are.
I need this. Take this.

15. EXPECT THESE TASKS

16. INPUT
(DATA)
=YOU+ OUTPUT
(VIS)

17. INPUT
(DATA)
=YOU+ OUTPUT
(VIS)
+
Get data
& Wrangle
1
+
Analyze
& Visualize
2

18. GET DATA & WRANGLE1

19. DATA SOURCES
Open data
Publicly available
Internal data
Private, owned by clients’ organization
Self-collected data
Manual, site scraping, etc.
Combine the above

20. MANY FORMS OF DATA
Standalone files
txt, csv, tsv, json, Google Docs, …, pdf*
APIs
better quality with more overhead
Databases
doesn’t necessary mean they are organized
Big data
bigger pain

21. HAVING ALL TWEETS
How people think I feel.

22. How people think I feel. How I really feel.
HAVING ALL TWEETS

23. CHALLENGES
Get relevant Tweets
hashtag: #oscars
keywords: “spotlight” (movie name)
Too big
Need to aggregate & reduce size
Slow
Long processing time (hours)

24. Hadoop Cluster
GETTING BIG DATA
Data Storage

25. Pig / Scalding (slow)
GETTING BIG DATA
Hadoop Cluster
Data Storage
Tool

26. Hadoop Cluster
Pig / Scalding (slow)
GETTING BIG DATA
Data Storage
Tool

27. Pig / Scalding (slow)
GETTING BIG DATA
Hadoop Cluster
Data Storage
Tool
Your laptop Smaller dataset

28. Hadoop Cluster
Pig / Scalding (slow)
Data Storage
Tool
Final dataset
Tool node.js / python / excel (fast)
Your laptop
GETTING BIG DATA
Smaller dataset

29. EXPECT TO WAIT FOR (BIG) DATA

30. DATA WRANGLING
Clean
A clean dataset? Joking, right?
Filter
Less is more
Parse, Format, Correct, etc.
Change country code from 3-letter to 2-letter
Correct time of day based on users’ timezone
etc.

31. EXPECT A LOT OF TIME
WITH DATA WRANGLING
70-80% of time
“Data Janitor”

32. RECOMMENDATIONS
Always think that you will have to do it again
document the process, automation
Reusable scripts
break a gigantic do-it-all function into smaller ones
Reusable data
keep for future project

33. ANALYZE & VISUALIZE2

34. EXPECT DIFFERENT REQUIREMENTS

35. TYPE OF PROJECTS
Explanatory Exploratory
Storytelling Analytics Tools Inspirations
x x
PMs, Data ScientistsGeneral Public General Public
Understand
product usage
See what data
can tell us
Get inspired

36. TYPE OF PROJECTS
Explanatory Exploratory
Storytelling Analytics Tools Inspirations
x x
PMs, Data ScientistsGeneral Public General Public
Understand
product usage
See what data
can tell us
Get inspired

37. So many things
we could learn
from Twitter data

38. Give us interesting vis
about xxxx by Nov 10

39. STORYTELLING : WHAT TO EXPECT
timely
Deadline is strict. Also can be unexpected events.
wide audience
easy to explain and understand, multi-device support
one-off projects
content screening

40. WHO/WHAT
STORYTELLING
WHERE WHEN
location time
user/content

41. WHO/WHAT
STORYTELLING
WHERE WHEN
location time
user/content

42. TIME : TWEETS/SECOND
by Miguel Rios

43. TIME : TWEETS/SECOND
by Miguel Rios

44. TIME : TWEETS/SECOND + ANNOTATION
http://www.flickr.com/photos/twitteroffice/5681263084/
by Miguel Rios

45. IT DOESN’T HAVE TO BE COMPLEX.

46. WHO/WHAT
STORYTELLING
WHERE WHEN
location time
user/content

47. LOCATION
Low density
High density
by Miguel Rios

48. LOCATION
flickr.com/photos/twitteroffice/8798020541
San Francisco
Low density
High density
by Miguel Rios

49. Rebuild the world
based on
tweet density
twitter.github.io/interactive/andes/
by Nicolas Garcia Belmonte

50. WHO/WHAT
STORYTELLING
WHERE WHEN
location time
user/content

51. CONTENT : US ELECTION 2016

52. CONTENT : #MUSEUMWEEK

53. CONTENT : #MUSEUMWEEK

54. WHO/WHAT
STORYTELLING
WHERE WHEN
location time
user/content

55. TIME + LOCATION : TWEET TIME BY CITY
Night
Late night
Daytime
Night
Late night
Daytime
by Miguel Rios & Jimmy Lin

56. Night
Late night
Daytime
Night
Late night
Daytime
TIME + LOCATION : TWEET TIME BY CITY
by Miguel Rios & Jimmy Lin

57. Night
Late night
Daytime
Night
Late night
Daytime
TIME + LOCATION : TWEET TIME BY CITY
by Miguel Rios & Jimmy Lin

58. TIME + LOCATION : TWEET TIME BY CITY
Night
Late night
Daytime
Night
Late night
Daytime
by Miguel Rios & Jimmy Lin

59. WHO/WHAT
STORYTELLING
WHERE WHEN
location time
user/content

60. CONTENT + LOCATION : TWEET MAP
by Robert Harris

61. CONTENT + LOCATION : TWEET MAP
by Robert Harris

62. most
frequent
term
CONTENT + LOCATION : TWEET MAP
by Robert Harris

63. Gmail was down
Jan 24, 2014
CONTENT + LOCATION : TWEET MAP
by Robert Harris

64. USER + LOCATION : FAN MAP
interactive.twitter.com/nfl_followers2014

65. USER + LOCATION : FAN MAP
interactive.twitter.com/nba_followers

66. USER + LOCATION : FAN MAP
interactive.twitter.com/premierleague

67. WHO/WHAT
STORYTELLING
WHERE WHEN
location time
user/content

68. CONTENT + TIME : STREAMGRAPH

69. CONTENT + TIME : MATCH SUMMARY
Biggest tournament for European soccer clubs

70. CONTENT + TIME : MATCH SUMMARY
Count Tweets mentioning the teams every minute
Dortmund Bayern Munich
Team 1 Team 2
time
begin
end

71. CONTENT + TIME : MATCH SUMMARY

72. CONTENT + TIME : MATCH SUMMARY
+ goals

73. CONTENT + TIME : MATCH SUMMARY
+ goals
+ players

74. CONTENT + TIME : COMPETITION SUMMARY
A B C D
A C
C
vs vs
vs + =
uclfinal.twitter.com

75. WHO/WHAT
STORYTELLING
WHERE WHEN
location time
user/content

76. CONTENT + TIME + LOCATION : NEW YEAR 2014
twitter.github.io/interactive/newyear2014/

77. BEHIND THE SCENE

78. https://interactive.twitter.com/tenyears
Project / Twitter 10 years

79. REQUEST

80. EXPECT FUNNY REQUESTS

81. DESIGN & PROTOTYPE
Engagements
First Minute First Hour First Day First Week
0 24h 0 7d0 60s 0 60m

83. EXPECT REVISIONS

87. Visualization is an important piece, but not the entire experience.
DON’T FORGET THE BIG PICTURE.

90. https://interactive.twitter.com/tenyears
Demo / Twitter 10 years

91. WORKFLOW
Requested / Identify needs
Design & Prototype
Refine
Mobile, Embed
Logging
Release

92. EXPECT THE UNEXPECTED

93. WORKFLOW
Requested / Identify needs
Design & Prototype
Refine
Mobile, Embed
Logging
Translations
Release

94. TYPE OF PROJECTS
Explanatory Exploratory
Storytelling Analytics Tools Inspirations
x x
PMs, Data ScientistsGeneral Public General Public
Understand
product usage
See what data
can tell us
Get inspired

95. Data sources
Output
explore
analyze
present
get
*
*

96. Data sources
Output
explore
analyze
present
get
*
*
ad-hoc scripts

97. Data sources
Output
explore
analyze
present
get
*
*
ad-hoc scripts tools for exploration

98. ANALYTICS TOOLS : WHAT TO EXPECT
richer, more features
to support exploration of complex data
more technical audience
product managers, engineers, data scientists
accuracy
designed for dynamic input
long-term projects

99. USER ACTIVITY LOGS

100. UsersUseTwitter

101. UsersUse
Product Managers
Curious
Twitter

102. UsersUse
Curious
Engineers
Log data
in Hadoop
Write Twitter
Instrument
Product Managers

103. WHAT ARE BEING LOGGED?
tweet
activities

104. WHAT ARE BEING LOGGED?
tweet from home timeline on twitter.com
tweet from search page on iPhone
activities

105. WHAT ARE BEING LOGGED?
tweet from home timeline on twitter.com
tweet from search page on iPhone
sign up
log in
retweet
etc.
activities

106. ORGANIZE?

107. LOG EVENT A.K.A. “CLIENT EVENT”
[Lee et al. 2012]

108. LOG EVENT A.K.A. “CLIENT EVENT”
client : page : section : component : element : action
web : home : timeline : tweet_box : button : tweet
1) User ID
2) Timestamp
3) Event name
4) Event detail
[Lee et al. 2012]

109. LOG DATA

110. UsersUse
Curious
Engineers
Log data
in Hadoop
Twitter
Instrument
Write
Product Managers
bigger than
Tweet data

111. UsersUse
Curious
Engineers
Log data
in Hadoop
Data Scientists
Ask
Twitter
Instrument
Write
Product Managers

112. UsersUse
Curious
Engineers
Log data
in Hadoop
Data Scientists
Find
Ask
Twitter
Instrument
Write
Product Managers

113. LOG DATA

114. UsersUse
Curious
Engineers
Log data
in Hadoop
Data Scientists
Find, Clean
Ask
Twitter
Instrument
Write
Product Managers

115. UsersUse
Curious
Engineers
Log data
in Hadoop
Data Scientists
Find, Clean
Ask
Monitor
Twitter
Instrument
Write
Product Managers

116. UsersUse
Curious
Engineers
Log data
in Hadoop
Data Scientists
Find, Clean, Analyze
Ask
Monitor
Twitter
Instrument
Write
Product Managers

117. Log data
EngineersData Scientists
Usersin Hadoop
Find, Clean, Analyze
Use
Monitor
Ask
Curious
1 2
Twitter
Instrument
Write
Product Managers

118. Scribe Radar
Project / Find & Monitor client events

119. Log data
in Hadoop
Engineers & Data Scientists
billions of rows

120. Log data
in Hadoop
Aggregate
Client events count
Engineers & Data Scientists

121. Log data
in Hadoop
Aggregate
Find
client page section component element action
Search
Client events count
Engineers & Data Scientists

122. Log data
in Hadoop
Aggregate
Find
client page section component element action
Search
Client events count
Engineers & Data Scientists

123. SECTION?
COMPONENT?
ELEMENT?

124. client page section component element action
Search
Find
Log data
in Hadoop
Aggregate
web home * * impression*
Client events count
Engineers & Data Scientists

125. client page section component element action
Search
Find
Query
Return
Log data
in Hadoop
Results
web : home : home : - : - : impression
web : home : wtf : - : - : impression
Aggregate
web home * * impression*
Client events count
Engineers & Data Scientists

126. client page section component element action
Search
Find
Query
Return
Log data
in Hadoop
Results
web : home : home : - : - : impression
web : home : wtf : - : - : impression
Aggregate
search can be better
Client events count
Engineers & Data Scientists

127. client page section component element action
Search
Find
Query
Return
Log data
in Hadoop
Results
web : home : home : - : - : impression
web : home : wtf : - : - : impression
Aggregate
10,000+ event types
search can be better
Client events count
Engineers & Data Scientists

128. client page section component element action
Search
Find
Query
Return
Log data
in Hadoop
Results
web : home : home : - : - : impression
web : home : wtf : - : - : impression
Aggregate
search can be better
10,000+ event types
not everybody knows
What are all sections under web:home?
Client events count
Engineers & Data Scientists

129. client page section component element action
Search
Find
Query
Return
Log data
in Hadoop
Results
web : home : home : - : - : impression
Aggregate
one graph / event
10,000+ event types
not everybody knows
What are all sections under web:home?
Client events count
Engineers & Data Scientists
search can be better

130. client page section component element action
Search
Find
Query
Return
Log data
in Hadoop
Results
web : home : home : - : - : impression
Aggregate
one graph / event
x 10,000
10,000+ event types
not everybody knows
What are all sections under web:home?
Client events count
Engineers & Data Scientists
search can be better

131. GOALS
Search for client events
Explore client event collection
Monitor changes

132. DESIGN

133. Client event collection
Engineers & Data Scientists

134. See
Client event collection
Engineers & Data Scientists

135. See
Client event collection
Engineers & Data Scientists
narrow down
Interactions
search box => filter

136. See
HOW TO VISUALIZE?
narrow down
Client event collection
Engineers & Data Scientists
Interactions
search box => filter

137. See
Client event collection
Engineers & Data Scientists
client : page : section : component : element : action
HOW TO VISUALIZE?
narrow down
Interactions
search box => filter

138. CLIENT EVENT HIERARCHY
iphone home -
- - impression
tweet tweet click
iphone:home:-:-:-:impression
iphone:home:-:tweet:tweet:click

139. DETECT CHANGES
iphone home -
- - impression
tweet tweet click
iphone home -
- - impression
tweet tweet click
TODAY
7 DAYS AGO
compared to

140. CALCULATE CHANGES
+5% +5% +5%
+10% +10% +10%
-5% -5% -5%
DIFF

141. DISPLAY CHANGES
iphone home -
- - impression
tweet tweet click
Map of the Market [Wattenberg 1999], StemView [Guerra-Gomez et al. 2013]

142. DISPLAY CHANGES
home -
- - impression
tweet tweet click
iphone

143. Demo Demo Demo
Demo / Scribe Radar

144. Twitter for Banana

146. Flying Sessions
Project / Funnel Analysis

147. COUNT PAGE VISITS
banana : home : - : - : - : impression
home page

148. FUNNEL
home page
profile page

149. FUNNEL ANALYSIS
1 jobhome page
profile page
1 hourbanana : home : - : - : - : impression
banana : profile : - : - : - : impression

150. FUNNEL ANALYSIS
banana : home : - : - : - : impression
banana : profile : - : - : - : impression banana : search : - : - : - : impression
home page
profile page search page
2 jobs
2 hours

151. FUNNEL ANALYSIS
banana : home : - : - : - : impression
banana : profile : - : - : - : impression banana : search : - : - : - : impression
home page
profile page search page
Specify all funnels manually!
n jobs
Time to find a
new job

152. GOAL
banana : home : - : - : - : impression
… ……
1 job => all funnels, visualized
home page

153. USER SESSIONS
Session#1
A
B
end
Session#4
Start
end
A
Session#2
B
end
A
Session#3
C
end
A
StartStartStart

154. AGGREGATE
A
BB C
Start
end endend
A A
end
A
4 sessions

155. AGGREGATE
A
BB C
Start
end endend
end
4 sessions

156. AGGREGATE
C
Start
end endend
end
A
B
4 sessions

157. AGGREGATE
C
Start
end endend
end
A
B
4 sessions

158. AGGREGATE
C
Start
end endend
A
B end
4 sessions

159. AGGREGATE
C
Start
endend
A
B end
4 sessions

160. AGGREGATE
C
Start
endend
A
B end
4 sessions

161. AGGREGATE
Start
endend
A
CB end
4 sessions

162. AGGREGATE
endend
A
CB end
Start
4,000,000 sessions

163. (~millions sessions, 10,000+ event types)
TRY WITH SAMPLE DATA

164. FAIL…

165. Keep trying to make it work
EXPECT TRIALS AND ERRORS

166. Read the details in
Krist Wongsuphasawat and Jimmy Lin.
“Using Visualizations to Monitor Changes and Harvest Insights from a Global-Scale Logging Infrastructure at Twitter “
Proc. IEEE Conference on Visual Analytics Science and Technology (VAST) 2014
HOW TO MAKE IT WORK?

167. Demo Demo Demo
Demo / Flying Sessions

168. WORKFLOW
Requested / Identify needs
Design & Prototype
Make it work for sample dataset
Refine & Generalize
Productionize
Document & Release
Maintain & Support
Keep it running, Feature requests & Bugs fix

169. TYPE OF PROJECTS
Explanatory Exploratory
Storytelling Analytics Tools Inspirations
x x
PMs, Data ScientistsGeneral Public General Public
Understand
product usage
See what data
can tell us
Get inspired

170. https://medium.com/@kristw/designing-the-game-of-tweets-7f87c30dc5a2
Project / Game of Tweets

173. EXPECT HARDWARE COMPLICATIONS

174. INPUT
(DATA)
=YOU+ OUTPUT
(VIS)
+
Get data
& Wrangle
1
+
Analyze
& Visualize
2

175. INPUT
(DATA)
=YOU+ OUTPUT
(VIS)
+
Get data
& Wrangle
1
+
Analyze
& Visualize
2

176. EXPECT TO IMPROVE

177. HOW TO BE BETTER?
Time is limited.
Grow the team
Expand skills
Improve tooling
Solve a problem once and for all
Automate repetitive tasks

178. http://twitter.github.io/labella.js
Demo / Labella.js

179. https://github.com/twitter/d3kit
Demo / d3Kit
http://www.slideshare.net/kristw/d3kit

180. yeoman.io
Demo / Yeoman

181. TO SUM UP

182. INPUT
(DATA)
=YOU+ OUTPUT
(VIS)
+
Get data
& Wrangle
1
+
Analyze
& Visualize
2

183. TYPE OF PROJECTS
Explanatory Exploratory
Storytelling Analytics Tools Inspirations
x x
PMs, Data ScientistsGeneral Public General Public
Understand
product usage
See what data
can tell us
Get inspired

184. TAKE-AWAY
Getting data and data wrangling are time-consuming.
Different projects, different requirements
Storytelling, Product insights, Art, etc.
Combine visualization with other skills
HCI, Design, Stats, ML, etc.
Expect the unexpected
Learn and improve
do more with less time
grow the team, expand skills, improve tooling
Krist Wongsuphasawat / @kristw
kristw.yellowpigz.com

185. Nicolas Garcia Belmonte, Robert Harris, Miguel Rios,
Simon Rogers, Jimmy Lin, Linus Lee, Chuang Liu,
and many colleagues at Twitter.
Lastly, to my wife for taking care of our 3 months old baby, so I had time to prepare these slides.
ACKNOWLEDGEMENT

186. RESOURCES
Images
Banana phone http://goo.gl/GmcMPq
Bar chart https://goo.gl/1G1GBg
Boss https://goo.gl/gcY8Kw
Champions League http://goo.gl/DjtNKE
Database http://goo.gl/5N7zZz
Fishing shark http://goo.gl/2fp4zW
Globe visualization http://goo.gl/UiGMMj
Harry Potter http://goo.gl/Q9Cy64
Holding phone http://goo.gl/It2TzH
Kiwi orange http://goo.gl/ejQ73y
Kiwi http://goo.gl/9yk7o5
Library https://goo.gl/HVeE6h
Library earthquake http://goo.gl/rBqBrs
Minion http://goo.gl/I19Ijg
NBA http://goo.gl/p7HBdG
NFL http://goo.gl/feQMZs
Orange & Apple http://goo.gl/NG6RIL
Pile of paper http://goo.gl/mGLQTx
Premier League http://goo.gl/AqIINO
Scrooge McDuck https://goo.gl/aKv8D7
The Sound of Music https://goo.gl/dqHlzj
Trash pile http://goo.gl/OsFfo3
Tyrion http://goo.gl/WaBonl
Watercolor Map by Stamen Design

187. THANK YOU