visualization at Twitter data Krist
Wongsuphasawat / @kristw

Krist Wongsuphasawat / @kristw

Bangkok, Thailand Krist Wongsuphasawat /
@kristw

Computer Engineer Bangkok, Thailand Chulalongkorn
University Krist Wongsuphasawat / @kristw

Computer Engineer Bangkok, Thailand Programming
+ Soccer Krist Wongsuphasawat / @kristw

Computer Engineer Bangkok, Thailand Programming
+ Soccer Krist Wongsuphasawat / @kristw

Computer Engineer Bangkok, Thailand Programming
+ Soccer Krist Wongsuphasawat / @kristw

Computer Engineer Bangkok, Thailand M.S.
in Computer Science Univ. of Maryland Krist Wongsuphasawat / @kristw

Computer Engineer Bangkok, Thailand PhD
in Computer Science Univ. of Maryland Information Visualization Krist Wongsuphasawat / @kristw

Computer Engineer Bangkok, Thailand PhD
in Computer Science Univ. of Maryland Information Visualization Krist Wongsuphasawat / @kristw IBM Microsoft

Computer Engineer Bangkok, Thailand PhD
in Computer Science Univ. of Maryland Information Visualization Krist Wongsuphasawat / @kristw IBM Microsoft Sr. Data Visualization Scientist Twitter

data visualization
at Twitter

data visualization
at Twitter

visualization
data
at Twitter

vis
data
at Twitter

data
at Twitter
“Tweets”

data at Twitter “Tweets” #events
TV Shows New Year Earthquake Oscars Protest Super Bowl World Cup Election Breaking news …

data at Twitter “Tweets” #events
TV Shows New Year Earthquake Oscars Protest Super Bowl World Cup Election Breaking news … #curiosity Sleep pattern Human behavior Language …

data at Twitter “Tweets” #events
TV Shows New Year Earthquake Oscars Protest Super Bowl World Cup Election Breaking news … #curiosity Sleep pattern Human behavior Language … What could we learn from the Tweets?

vis data at Twitter “Tweets”
Tell stories about an event, Pursue curiosity or inspiration Goal:

vis data at Twitter “Tweets”
Tell stories about an event, Pursue curiosity or inspiration (with deadline) Goal:

Challenge accepted

vis data at Twitter “Tweets”
Get data 1

easy?

Having all Tweets How people
think I feel.

Having all Tweets How people
think I feel. How I really feel.

Challenges • Too much data
• Want only relevant Tweets • hashtag: #BRA • keywords: “goal” • Need to aggregate & reduce size • Long processing time (hours)

Hadoop Cluster Vertica Pig /
Scalding (slow) SQL Data Storage Tool Workflow

Hadoop Cluster Vertica Pig /
Scalding (slow) SQL Data Storage Tool Workflow

Workflow Hadoop Cluster Vertica Pig
/ Scalding (slow) SQL Data Storage Tool Your laptop Smaller dataset

Hadoop Cluster Vertica Pig /
Scalding (slow) SQL Data Storage Tool Tool node.js / python / excel (fast) Final dataset Your laptop Workflow Smaller dataset

vis data at Twitter “Tweets”
Get data 1 2 Visualize

Visualize • Peek into data
• Check data & test ideas • Decide how to visualize • Guided by data type • Choose tools • Start building

Visualize • Peek into data
• Check data & test ideas • Decide how to visualize • Guided by data type • Choose tools • Start building R d3 Tableau Yeoman

(+ media) photos, videos data
What? TEXT Where? When? GEO TIME

Visualize Data What? TEXT Where?
When? GEO TIME

Visualize Data What? TEXT Where?
When? GEO TIME

Time Tweets/second

Time Tweets/second

Time Tweets/second + Annotation
http://www.flickr.com/photos/twitteroffice/5681263084/

Visualize Data What? TEXT Where?
When? GEO TIME

Geo Heatmap Low density High
density

Geo San Francisco flickr.com/photos/twitteroffice/8798020541 Low
density High density

Geo San Francisco Rebuild the
world based on tweet volumes twitter.github.io/interactive/andes/

Visualize Data What? TEXT Where?
When? GEO TIME

Text www.wordle.net Some experiments during
World Cup

Text www.wordle.net Word cloud of
Tweets right after the 1st goal

Text Word cloud of Tweets
right after the 1st goal It was an “own” goal. www.wordle.net

Text WordTree [Wattenberg & Viégas
2008] www.jasondavies.com/wordtree www.jasondavies.com/wordtree

Visualize Data What? TEXT Where?
When? GEO TIME

Time + Geo Japan Earthquake
2011 blog.twitter.com/2011/global-pulse youtu.be/SybWjN9pKQk

Time + Geo Tweet pattern
[Rios & Lin 2012] Night Late night Daytime Night Late night Daytime

Time + Geo Tweet pattern
[Rios & Lin 2012] Night Late night Daytime Night Late night Daytime

Time + Geo Tweet pattern
[Rios & Lin 2012] Night Late night Daytime Night Late night Daytime

Time + Geo Tweet pattern
[Rios & Lin 2012] Night Late night Daytime Night Late night Daytime

Visualize Data What? TEXT Where?
When? GEO TIME

Geo + Text Real-time Tweet
map

Geo + Text Real-time Tweet
map

Geo + Text Real-time Tweet
map most frequent term

Geo + Text Real-time Tweet
map Gmail was down Jan 24, 2014

Geo + Text Real-time Tweet
map Nelson Mandela passed away Dec 5, 2013

Visualize Data What? TEXT Where?
When? GEO TIME

Time + Text UEFA Champions
League Biggest tournament for European soccer clubs Many Tweets during the matches

UEFA Champions League Team 1
Team 2 Time + Text Dortmund Bayern Munich

UEFA Champions League Team 1
Team 2 Time + Text Dortmund Bayern Munich

UEFA Champions League Team 1
Team 2 Time + Text Dortmund Bayern Munich

UEFA Champions League Team 1
Team 2 Dortmund Bayern Munich Count Tweets mentioning the teams every minute Time + Text

Time + Text UEFA Champions
League

Time + Text UEFA Champions
League + “goal” count + context

+ “offside” Time + Text
UEFA Champions League

+ players Time + Text
UEFA Champions League

Competition Tree vs vs A
B C D vs A C C

Competition Tree vs vs A
B C D vs + A C C

Competition Tree vs vs A
B C D vs + = A C C

Visualize Data What? TEXT Where?
When? GEO TIME

Time + Text + Geo
State of the Union twitter.github.io/interactive/sotu2014

Time + Text + Geo
State of the Union 1) timeline + topic from Tweets 4) Density map of Tweets about selected topic 3) Volume of Tweets by topics during selected part of the SOTU 2) context (speech) twitter.github.io/interactive/sotu2014

Time + Text World Cup
2014

Time + Text + Geo
World Cup 2014

Visualize Data What? TEXT Where?
When? GEO TIME

Visualize Data What? TEXT +
Where? When? GEO TIME Non-Twitter data CONTEXT

Time + Text New Year
2014

Time + Text New Year
2014

Time + Text + Geo
(c) New Year 2014 twitter.github.io/interactive/newyear2014/

vis data at Twitter “Tweets”
Get data 1 2 Visualize

vis data at Twitter “Tweets”
Get data 1 2 Visualize Evaluate 3

vis data at Twitter “Tweets”
Get data 1 2 Visualize Evaluate 3 Iterate!

Evaluation • Self • Peer
feedback • Non team members / Potential audience

vis data at Twitter Get
data 1 2 Visualize Evaluate 3

vis data at Twitter Get
data 1 2 Visualize Evaluate 3 big data => small data

vis data at Twitter Get
data 1 2 Visualize Evaluate 3 big data => small data What? Where? When?

big data => small data
self, peer, external vis data at Twitter Get data 1 2 Visualize Evaluate 3 What? Where? When?

big data => small data
self, peer, external vis data at Twitter “Tweets” Get data 1 2 Visualize Evaluate 3 What? Where? When?

big data => small data
self, peer, external vis data at Twitter “Tweets” Get data 1 2 Visualize Evaluate 3 What? Where? When? • users • followers graph • logs • etc. ! • derived data: language, sentiment

big data => small data
self, peer, external vis data at Twitter “Tweets” Get data 1 2 Visualize Evaluate 3 What? Where? When? • users Who? … • followers graph • logs • etc. ! • derived data: language, sentiment

big data => small data
self, peer, external vis data at Twitter “Tweets” Get data 1 2 Visualize Evaluate 3 What? Where? When? • users Who? … • followers graph • logs • etc. ! • derived data: language, sentiment (with deadline)

big data => small data
self, peer, external vis data at Twitter “Tweets” Get data 1 2 Visualize Evaluate 3 What? Where? When? • users Who? … • followers graph • logs • etc. (with deadline) ! • derived data: language, sentiment @kristw / https://interactive.twitter.com

big data => small data
self, peer, external vis data at Twitter “Tweets” Get data 1 2 Visualize Evaluate 3 What? Where? When? • users Who? … • followers graph • logs • etc. (with deadline) ! • derived data: language, sentiment @kristw / https://interactive.twitter.com + visualizations by @philogb, @miguelrios & @trebor

Questions?

big data => small data
self, peer, external vis data at Twitter “Tweets” Get data 1 2 Visualize Evaluate 3 What? Where? When? • users Who? … • followers graph • logs • etc. (with deadline) @kristw / https://interactive.twitter.com + visualizations by @philogb, @miguelrios & @trebor

Thank you