1. Krist Wongsuphasawat
@kristw
WHAT I TELL MYSELF
BEFORE
VISUALIZING

2. WHAT I TELL MYSELF
BEFORE
VISUALIZING
Krist Wongsuphasawat
@kristw

3. Computer Engineer
Bangkok, Thailand
Chulalongkorn University
Krist Wongsuphasawat / @kristw

4. Programming & Football
Computer Engineer
Bangkok, Thailand
Krist Wongsuphasawat / @kristw

5. Krist Wongsuphasawat / @kristw
Programming & Football
Computer Engineer
Bangkok, Thailand

6. (P.S. These are actually not my robots, but our competitors’.)
Krist Wongsuphasawat / @kristw
Computer Engineer
Bangkok, Thailand

7. PhD in Computer Science
Information Visualization
Univ. of Maryland
Krist Wongsuphasawat / @kristw
Computer Engineer
Bangkok, Thailand

8. IBM
Data Scientist
Analytics, Experiment
Twitter
Microsoft
Krist Wongsuphasawat / @kristw
PhD in Computer Science
Information Visualization
Univ. of Maryland
Computer Engineer
Bangkok, Thailand

9. IBM
Krist Wongsuphasawat / @kristw
Engineering Manager
Data Experience
Airbnb
Microsoft
Twitter
PhD in Computer Science
Information Visualization
Univ. of Maryland
Computer Engineer
Bangkok, Thailand

10. Public-facing visualizations
Open-source projects
Visual Analytics Tools
interactive.twitter.com
Apache Superset (31,000+ ⭐)
labella.js (3000+ ⭐)
visx (10000+ ⭐)
react-vega, encodable
Internal tools
Academic papers
kristw.yellowpigz.com

11. DATA =
ME
+ VIS

12. Data, I’m ready!

13. Data, I’m ready!
Here I come!

14. WHAT TO EXPECT?

15. 1.
EXPECT TO
FIND THE
REAL NEED

16. INPUT (DATA)
What clients think they have

17. INPUT (DATA)
What clients think they have What they usually have

18. YOU
What clients think you are

19. YOU
What clients think you are What they will get

20. OUTPUT (VIS)
What clients ask for

21. OUTPUT (VIS)
What clients ask for What they really need

22. COMMUNICATE

23. GOALS
Present data
Communicate information effectively
Analyze data
Exploratory data analysis
Tools to analyze data
Reusable tools for exploration
Enjoy
Combination of above

24. GOALS
Present data
Communicate information effectively
Analyze data
Exploratory data analysis
Tools to analyze data
Reusable tools for exploration
Enjoy
Combination of above
Who are the audience?
What do you want to tell?
What are the questions?
Who will use this?
What would they use this for?
Who are the audience?

25. I need this. Take this.

26. I need this. Here you are.
I need this. Take this.

27. & COMPROMISE

28. 2.
EXPECT TO
CLEAN DATA

29. 2.
EXPECT TO
CLEAN DATA
A LOT

30. 70-80% of time cleaning data
“DATA JANITOR”

31. Collect + Clean + Transform
DATA WRANGLING

32. WHY DOES IT TAKE
SO MUCH TIME?

33. 2.1 Many sources and formats

34. DATA SOURCES
Open data
Publicly available
Internal data
Private, owned by clients’ organization
Self-collected data
Manual, site scraping, etc.
Combine the above

35. DATA FORMAT
Standalone files
txt, csv, tsv, json, Google Docs, …, pdf*
Databases
doesn’t necessary mean they are organized
API
better quality with more overhead
Big data*

36. NEED TO…
Change format
e.g. tsv => json
Combine data
Resolve multiple sources of truth

37. 2.2 Data transformation

38. EXAMPLES
Convert latitude/longitude into country
Change country code from 3-letter (USA) to 2-letter (US)
Correct time of day based on users’ timezone
etc.

39. 2.3 Data collection issues

40. EXAMPLES
Typos
Incorrect values
Incorrect timestamps
Missing data

41. 2.4 Definition of “clean” data

42. IS THIS CLEAN?
USER RESTAURANT RATING
========================
A MCDONALD’S 3
B MCDONALDS 3
C MCDONALD 4
D MCDONALDS 5
E IHOP 4
F SUBWAY 4

43. IS THIS CLEAN?
USER RESTAURANT RATING
========================
A MCDONALD’S 3
B MCDONALDS 3
C MCDONALD 4
D MCDONALDS 5
E IHOP 4
F SUBWAY 4
How many reviews are there?
Clean.
How many restaurants are there?
Not clean.
McDonald, McDonald’s, McDonalds

44. 2.5 Bigger data, bigger problems

45. HAVING ALL TWEETS
How people think I feel.

46. How people think I feel. How I really feel.
HAVING ALL TWEETS

47. Lots of machines
GETTING BIG DATA
Data Warehouse

48. Spark, Hadoop, etc. (slow)
GETTING BIG DATA
Tool
Lots of machines
Data Warehouse

49. GETTING BIG DATA
Tool
Your laptop Smaller dataset
Spark, Hadoop, etc. (slow)
Lots of machines
Data Warehouse

50. Tool
Final dataset
Tool node.js / python / excel (fast)
Your laptop
GETTING BIG DATA
Smaller dataset
Spark, Hadoop, etc. (slow)
Lots of machines
Data Warehouse

51. CHALLENGES
Slow
Long processing time (hours)
Get relevant Tweets
keywords: “parasite” (movie name)
Too big
Need to aggregate & reduce size
Harder to spot problems

52. CHALLENGES
Slow
Long processing time (hours)
Get relevant Tweets
keywords: “parasite” (movie name)
Too big
Need to aggregate & reduce size
Harder to spot problems

53. 2.6 Issues can show up any time.

55. RECOMMENDATIONS
Always think that you will have to do it again
document the process, automation
Reusable scripts
break large script into smaller ones
Reusable data
keep for future project

56. 3.
PREPARE
TO ITERATE
AGAIN & AGAIN

57. It was a great idea … until I actually tried it.

58. Celebrate your failures
#D3BrokeAndMadeArt
https://twitter.com/enjalot/status/1313159226995466240?s=20

59. TIPS
Don’t give up.
If stuck, take a break. Look for inspirations.
The vis that gives you insights may or may not be the vis for sharing.
Exploration vs. Communication
Keep it as simple as possible
but not simpler.

60. “Necessity is the mother of invention.”
— Old proverb

61. “Necessity is the mother of invention.”
DEADLINE
— Old proverb

62. TIPS
Don’t give up.
If stuck, take a break. Look for inspirations.
The vis that gives you insights may or may not be the vis for sharing.
Exploration vs. Communication
Keep it as simple as possible
but not simpler.
Set deadlines

63. BOBA SCIENCE
PROJECT /

64. https://medium.com/s/story/boba-science-
how-can-i-drink-a-bubble-tea-to-ensure-that-
i-dont-finish-the-tea-before-the-
bobas-7fc5fd0e442d

65. LOTS OF ITERATIONS

68. GAME OF THRONES
PROJECT /

69. Reveal the talking points
of every episode of
from fans’ conversations

70. PROBLEM
Understand what the audience
talk about a TV show
from Tweets

71. HBO’S GAME OF THRONES
Based on a book series “A Song of Ice and Fire”
Medieval Fantasy. Knights, magic and dragons.

72. HBO’S GAME OF THRONES
Based on a book series “A Song of Ice and Fire”
Medieval Fantasy. Knights, magic and dragons.
Many characters.
Anybody can die.
8 seasons
Multiple storylines in each episode

73. IDEAS
Common words
Too much noise

74. IDEAS
Common words
Too much noise
Characters
How often each character were mentioned?

75. PROTOTYPING
Pull sample data
from Twitter API
Count characters
naive approach

76. LIST OF NAMES
Daenerys Targaryen,Khaleesi
Jon Snow
Sansa Stark
Tyrion Lannister
Arya Stark
Cersei Lannister
Khal Drogo
Gregor Clegane,Mountain
Margaery Tyrell
Joffrey Baratheon
Bran Stark
Theon Greyjoy
Jaime Lannister
Brienne
Eddard Stark,Ned Stark
Ramsay Bolton
Sandor Clegane,Hound
Ygritte
Stannis Baratheon
Petyr Baelish,Little Finger
Robb Stark
Bronn
Varys
Catelyn Stark
Oberyn Martell
Daario Naharis
Davos Seaworth
Jorah Mormont
Melisandre
Myrcella Baratheon
Tywin Lannister
Tommen Baratheon
Grey Worm
Tyene Sand
Rickon Stark
Missandei
Roose Bolton
Robert Baratheon
Jojen Reed
Jeor Mormont
Tormund Giantsbane
Lysa Arryn
Yara Greyjoy,Asha Greyjoy
Samwell Tarly,Sam
Hodor
Victarion Greyjoy
High Sparrow
Dragon
Winter
Dothraki

77. SAMPLE TWEET

78. SAMPLE TWEET

79. SAMPLE DATA
Character Count
Hodor 10000
Jon Snow 5000
Daenerys 4000
Bran Stark 3000
… …
*These numbers are made up for presentation, not real data.

80. WHERE TO GO FROM HERE?

81. + EMOTION

82. + CONNECTIONS

83. + CONNECTIONS

84. FOCUS ON
EMOTION & CONNECTIONS
WITHIN EPISODE

85. SAMPLE DATA
Character Count
Jon Snow+Sansa 1000
Tormund+Brienne 500
Bran Stark+Hodor 300
… …
Character Count
Hodor 10000
Jon Snow 5000
Daenerys 4000
… …
INDIVIDUALS CONNECTIONS
+ top emojis + top emojis
*These numbers are made up for presentation, not real data.

86. GRAPH
NODES LINKS
+ top emojis + top emojis
Character Count
Jon Snow+Sansa 1000
Tormund+Brienne 500
Bran Stark+Hodor 300
… …
Character Count
Hodor 10000
Jon Snow 5000
Daenerys 4000
… …
*These numbers are made up for presentation, not real data.

87. ISSUE: HAIRBALL

88. TRIED: MANUAL POSITIONS

89. + COLLISION DETECTION
http://blockbuilder.org/kristw/2850f65d6329c5fef6d5c9118f1de6e6

90. + COMMUNITY DETECTION
https://github.com/upphiminn/jLouvain

91. + COLLISION DETECTION (WITH CLUSTERS)
https://bl.ocks.org/mbostock/7881887

92. LET’S GET OTHER EPISODES.

93. MORE DATA
Hadoop
Rewrite the scripts
to get archived data

94. HOW MUCH DATA DO WE NEED?
Whole week?
5 days?
2 days?
A day?
etc.

95. HOW MUCH DATA DO WE NEED?

96. THE VIS IS NOT ENOUGH.

98. Legend

99. Navigation

100. Top 3

101. Adjust threshold

102. Recap

103. Filtered Recap
Tooltip

104. DEMO
https:/
/interactive.twitter.com/game-of-thrones

105. MOBILE SUPPORT

106. 4.
RESERVE
TIME FOR
REFINEMENT

107. “The first 90% of the code
accounts for the first 90% of the development time.
The remaining 10% of the code
accounts for the other 90% of the development time.”
— Tom Cargill, Bell Labs

108. REFINE & POLISH
Color
UX / UI + Mobile Support
Animation / Transition
Metadata for SEO
Social media preview images
Performance
Loading time, Data file size

109. FANDOM MAPS
PROJECT /

110. FAN MAP - NFL
interactive.twitter.com/

111. FAN MAP - NBA
interactive.twitter.com

112. FAN MAP - ENGLISH PREMIER LEAGUE
interactive.twitter.com

113. FAN MAP - ENGLISH PREMIER LEAGUE

114. INTERACTIVE.TWITTER.COM

115. GAME OF THRONES
PROJECT /

116. TRANSITIONS

117. CHANGING EPISODE (BEFORE)

118. CHANGING EPISODE (AFTER)

119. EXAMPLE

120. ISSUE: CONVEX HULL
http://bl.ocks.org/mbostock/4341699

121. X & Y ONLY, NO RADIUS

122. FIX IT

123. 5.
PLAN FOR
FEEDBACK

124. “Feedback is the breakfast of champion.”
— Ken Blanchard

125. FEEDBACK
During development
Feedback sessions with clients/potential users
After release
Logging
User study
Office hours

126. FEEDBACK

127. FEEDBACK

128. 6.
LOOK BACK
TO MOVE
FORWARD

129. WHAT COULD HAVE BEEN BETTER?
If I knew how to do XXX…
Learning opportunities
If I had someone who can do XXX…
Look for help
Grow the team
If I did not have to do the same tasks again…
Reusable components
Automate repetitive tasks

130. LABELLA.JS
PROJECT /

132. VISX = REACT + D3
PROJECT /

134. SUMMARY

135. WHAT I TELL MYSELF BEFORE VISUALIZING
1.
2.
3.
4.
5.
6.
Krist Wongsuphasawat / @kristw
kristw.yellowpigz.com
Expect to find the real need
Expect to clean data a lot
Prepare to iterate again & again
Reserve time for refinement
Plan for feedback
Look back to move forward

136. My former and current colleagues at Twitter and Airbnb
for their collaboration and support in these projects;
and my wife for taking care of our two kids
while I make these slides.
ACKNOWLEDGEMENT

137. WHAT I TELL MYSELF BEFORE VISUALIZING
1.
2.
3.
4.
5.
6.
Krist Wongsuphasawat / @kristw
kristw.yellowpigz.com
Expect to find the real need
Expect to clean data a lot
Prepare to iterate again & again
Reserve time for refinement
Plan for feedback
Look back to move forward

138. THANK YOU