Data Visualization at Twitter

visualization at Twitter data Krist Wongsuphasawat / @kristw

Krist Wongsuphasawat / @kristw

Bangkok, Thailand Krist Wongsuphasawat / @kristw

Computer Engineer Bangkok, Thailand Chulalongkorn University Krist Wongsuphasawat / @kristw

Computer Engineer Bangkok, Thailand Programming + Soccer Krist Wongsuphasawat / @kristw

Computer Engineer Bangkok, Thailand M.S. in Computer Science Univ. of Maryland Krist Wongsuphasawat / @kristw

Computer Engineer Bangkok, Thailand PhD in Computer Science Univ. of Maryland Information Visualization Krist Wongsuphasawat / @kristw

Computer Engineer Bangkok, Thailand PhD in Computer Science Univ. of Maryland Information Visualization Krist Wongsuphasawat / @kristw IBM Microsoft

Computer Engineer Bangkok, Thailand PhD in Computer Science Univ. of Maryland Information Visualization Krist Wongsuphasawat / @kristw IBM Microsoft Sr. Data Visualization Scientist Twitter

data visualization at Twitter

visualization data at Twitter

vis data at Twitter

data at Twitter “Tweets”

data at Twitter “Tweets” #events TV Shows New Year Earthquake Oscars Protest Super Bowl World Cup Election Breaking news …

data at Twitter “Tweets” #events TV Shows New Year Earthquake Oscars Protest Super Bowl World Cup Election Breaking news … #curiosity Sleep pattern Human behavior Language …

data at Twitter “Tweets” #events TV Shows New Year Earthquake Oscars Protest Super Bowl World Cup Election Breaking news … #curiosity Sleep pattern Human behavior Language … What could we learn from the Tweets?

vis data at Twitter “Tweets” Tell stories about an event, Pursue curiosity or inspiration Goal:

vis data at Twitter “Tweets” Tell stories about an event, Pursue curiosity or inspiration (with deadline) Goal:

Challenge accepted

vis data at Twitter “Tweets” Get data 1

easy?

Having all Tweets How people think I feel.

Having all Tweets How people think I feel. How I really feel.

Challenges • Too much data • Want only relevant Tweets • hashtag: #BRA • keywords: “goal” • Need to aggregate & reduce size • Long processing time (hours)

Hadoop Cluster Vertica Pig / Scalding (slow) SQL Data Storage Tool Workflow

Workflow Hadoop Cluster Vertica Pig / Scalding (slow) SQL Data Storage Tool Your laptop Smaller dataset

Hadoop Cluster Vertica Pig / Scalding (slow) SQL Data Storage Tool Tool node.js / python / excel (fast) Final dataset Your laptop Workflow Smaller dataset

vis data at Twitter “Tweets” Get data 1 2 Visualize

Visualize • Peek into data • Check data & test ideas • Decide how to visualize • Guided by data type • Choose tools • Start building

Visualize • Peek into data • Check data & test ideas • Decide how to visualize • Guided by data type • Choose tools • Start building R d3 Tableau Yeoman

(+ media) photos, videos data What? TEXT Where? When? GEO TIME

Visualize Data What? TEXT Where? When? GEO TIME

Time Tweets/second

Time Tweets/second + Annotation http://www.flickr.com/photos/twitteroffice/5681263084/

Geo Heatmap Low density High density

Geo San Francisco flickr.com/photos/twitteroffice/8798020541 Low density High density

Geo San Francisco Rebuild the world based on tweet volumes twitter.github.io/interactive/andes/

Text www.wordle.net Some experiments during World Cup

Text www.wordle.net Word cloud of Tweets right after the 1st goal

Text Word cloud of Tweets right after the 1st goal It was an “own” goal. www.wordle.net

Text WordTree [Wattenberg & Viégas 2008] www.jasondavies.com/wordtree www.jasondavies.com/wordtree

Time + Geo Japan Earthquake 2011 blog.twitter.com/2011/global-pulse youtu.be/SybWjN9pKQk

Time + Geo Tweet pattern [Rios & Lin 2012] Night Late night Daytime Night Late night Daytime

Geo + Text Real-time Tweet map

Geo + Text Real-time Tweet map most frequent term

Geo + Text Real-time Tweet map Gmail was down Jan 24, 2014

Geo + Text Real-time Tweet map Nelson Mandela passed away Dec 5, 2013

Time + Text UEFA Champions League Biggest tournament for European soccer clubs Many Tweets during the matches

UEFA Champions League Team 1 Team 2 Time + Text Dortmund Bayern Munich

UEFA Champions League Team 1 Team 2 Dortmund Bayern Munich Count Tweets mentioning the teams every minute Time + Text

Time + Text UEFA Champions League

Time + Text UEFA Champions League + “goal” count + context

+ “offside” Time + Text UEFA Champions League

+ players Time + Text UEFA Champions League

Competition Tree vs vs A B C D vs A C C

Competition Tree vs vs A B C D vs + A C C

Competition Tree vs vs A B C D vs + = A C C

Time + Text + Geo State of the Union twitter.github.io/interactive/sotu2014

Time + Text + Geo State of the Union 1) timeline + topic from Tweets 4) Density map of Tweets about selected topic 3) Volume of Tweets by topics during selected part of the SOTU 2) context (speech) twitter.github.io/interactive/sotu2014

Time + Text World Cup 2014

Time + Text + Geo World Cup 2014

Visualize Data What? TEXT + Where? When? GEO TIME Non-Twitter data CONTEXT

Time + Text New Year 2014

Time + Text + Geo (c) New Year 2014 twitter.github.io/interactive/newyear2014/

vis data at Twitter “Tweets” Get data 1 2 Visualize

vis data at Twitter “Tweets” Get data 1 2 Visualize Evaluate 3

vis data at Twitter “Tweets” Get data 1 2 Visualize Evaluate 3 Iterate!

Evaluation • Self • Peer feedback • Non team members / Potential audience

vis data at Twitter Get data 1 2 Visualize Evaluate 3

vis data at Twitter Get data 1 2 Visualize Evaluate 3 big data => small data

vis data at Twitter Get data 1 2 Visualize Evaluate 3 big data => small data What? Where? When?

big data => small data self, peer, external vis data at Twitter Get data 1 2 Visualize Evaluate 3 What? Where? When?

big data => small data self, peer, external vis data at Twitter “Tweets” Get data 1 2 Visualize Evaluate 3 What? Where? When?

big data => small data self, peer, external vis data at Twitter “Tweets” Get data 1 2 Visualize Evaluate 3 What? Where? When? • users • followers graph • logs • etc. ! • derived data: language, sentiment

big data => small data self, peer, external vis data at Twitter “Tweets” Get data 1 2 Visualize Evaluate 3 What? Where? When? • users Who? … • followers graph • logs • etc. ! • derived data: language, sentiment

big data => small data self, peer, external vis data at Twitter “Tweets” Get data 1 2 Visualize Evaluate 3 What? Where? When? • users Who? … • followers graph • logs • etc. ! • derived data: language, sentiment (with deadline)

big data => small data self, peer, external vis data at Twitter “Tweets” Get data 1 2 Visualize Evaluate 3 What? Where? When? • users Who? … • followers graph • logs • etc. (with deadline) ! • derived data: language, sentiment @kristw / https://interactive.twitter.com

big data => small data self, peer, external vis data at Twitter “Tweets” Get data 1 2 Visualize Evaluate 3 What? Where? When? • users Who? … • followers graph • logs • etc. (with deadline) ! • derived data: language, sentiment @kristw / https://interactive.twitter.com + visualizations by @philogb, @miguelrios & @trebor

Questions?

big data => small data self, peer, external vis data at Twitter “Tweets” Get data 1 2 Visualize Evaluate 3 What? Where? When? • users Who? … • followers graph • logs • etc. (with deadline) @kristw / https://interactive.twitter.com + visualizations by @philogb, @miguelrios & @trebor

Thank you