from fans’ conversations
Krist Wongsuphasawat / @kristw
Reveal the talking points
of every episode of

Computer Engineer
Bangkok, Thailand
PhD in Computer Science
Information Visualization
Univ. of Maryland
IBM
Microsoft
Data Visualization Scientist
Twitter
Krist Wongsuphasawat / @kristw

interactive.twitter.com open-source visualization visual analytics tools

Problem is coming.
CHAPTER I

“Problem first, not solution backward”
— Brian Caffo (via Ron Brookmeyer)

“If all you have is a hammer,
everything looks like a nail.”
— Abraham Maslow

Problem
Want to know what the audience
talk about a TV show

Problem
Want to know what the audience
talk about a TV show
from Tweets

HBO’s Game of Thrones
Based on a book series “A Song of Ice and Fire”
Medieval Fantasy. Knights, magic and dragons.

Brief Story

A King dies.
A lot of contenders wage a war
to reclaim the throne.

Minor characters with no claim to the throne
set their own plans in action to gain power
when all the major characters end up killing each other.

Brave/Honest/Honorable characters die.
Intelligent but shady characters
and characters who know nothing
continue to live.

On another continent,
there is a princess who was supposed to be
rightful heir to the throne.
The dead king mentioned earlier,
overthrew her father
and killed her entire family.
She escaped.
Now she is finding her way back
and she has dragons.

While humans are busy killing each other,
ice zombies “White walkers” are invading from the North.
The only group who seems to care about this
is neutral group called the Night’s Watch.

HBO’s Game of Thrones
Based on a book series “A Song of Ice and Fire”
Medieval Fantasy. Knights, magic and dragons.
Many characters.
Anybody can die.
6 seasons (57 episodes) so far
Multiple storylines in each episode

Problem
Want to know what the audience
talk about a TV show
from Tweets

Ideas
Common words
Too much noise

Ideas
Common words
Too much noise
Characters
How o"en each character were mentioned?

I demand a trial by prototyping.
CHAPTER II

Prototyping
Pull sample data
from Twitter API
Entity recognition and counting
naive approach

List of names
Daenerys Targaryen,Khaleesi
Jon Snow
Sansa Stark
Tyrion Lannister
Arya Stark
Cersei Lannister
Khal Drogo
Gregor Clegane,Mountain
Margaery Tyrell
Joffrey Baratheon
Bran Stark
Theon Greyjoy
Jaime Lannister
Brienne
Eddard Stark,Ned Stark
Ramsay Bolton
Sandor Clegane,Hound
Ygritte
Stannis Baratheon
Petyr Baelish,Little Finger
Robb Stark
Bronn
Varys
Catelyn Stark
Oberyn Martell
Daario Naharis
Davos Seaworth
Jorah Mormont
Melisandre
Myrcella Baratheon
Tywin Lannister
Tommen Baratheon
Grey Worm
Tyene Sand
Rickon Stark
Missandei
Roose Bolton
Robert Baratheon
Jojen Reed
Jeor Mormont
Tormund Giantsbane
Lysa Arryn
Yara Greyjoy,Asha Greyjoy
Samwell Tarly,Sam
Hodor
Victarion Greyjoy
High Sparrow
Dragon
Winter
Dothraki

Sample Tweet

Sample Tweet

Sample data
Character Count
Hodor 10000
Jon Snow 5000
Daenerys 4000
Bran Stark 3000
… …
*These numbers are made up for presentation, not real data.

When you play the game of vis,
you iterate or you die.
CHAPTER III

Where to go from here?

+ episodes
The Guardian & Google Trends
http://www.theguardian.com/news/datablog/ng-interactive/2016/apr/22/game-of-thrones-the-most-googled-characters-episode-by-episode

+ emotion

+ connections

+ connections

Gain insights from a single episode
emotion & connections

Better matching
Regular Expression
/gregor([ ]?clegane)?|(the mountain)/

Sample data
Character Count
Jon Snow+Sansa 1000
Tormund+Brienne 500
Bran Stark+Hodor 300
… …
Character Count
Hodor 10000
Jon Snow 5000
Daenerys 4000
… …
INDIVIDUALS CONNECTIONS
+ top emojis + top emojis
*These numbers are made up for presentation, not real data.

Graph
NODES LINKS
+ top emojis + top emojis
Character Count
Jon Snow+Sansa 1000
Tormund+Brienne 500
Bran Stark+Hodor 300
… …
Character Count
Hodor 1000
Jon Snow 500
Daenerys 400
… …
*These numbers are made up for presentation, not real data.

Network Visualization
Node-link diagram
Force-directed layout
http://blockbuilder.org/kristw/762b680690e4b2b2666dfec15838a384

Issue: Hairball

Why?
Too many nodes & edges
nodes = nodes.filter(n => n.count > 100)
links = links.filter(l => l.count > 100)
The force is (too) strong.
force
.charge(…)
.gravity(…)
.linkDistance(…)
.linkStrength(…)

Issue: Occlusions

Tried: Fixed positions

+ Collision Detection
http://blockbuilder.org/kristw/2850f65d6329c5fef6d5c9118f1de6e6

+ Community Detection
https://github.com/upphiminn/jLouvain

+ Collision Detection (with clusters)
https://bl.ocks.org/mbostock/7881887

Issue: Convex hull
http://bl.ocks.org/mbostock/4341699
d3.geom.hull(vertices)

x & y only, no radius

Example

Fix it

Fix it

Let’s get other episodes.

Hadoop remembers.
CHAPTER IV

More data
Hadoop
Rewrite the scripts in Scalding
to get archived data

How much data do we need?
Whole week?
5 days?
2 days?
A day?
etc.

How much data do we need?

Must work for every episode
Too many nodes & edges
nodes = nodes.filter(n => n.count > 100)
links = links.filter(l => l.count > 100)
Is 100 a good number for every episode?

Transitions

… Jumpy …

Too fast?
Slow down
force.friction(f)
0 = stop
0.1 = very slow
1 = very fast (frictionless)

A$er switching episode
1. Store old positions for existing objects.
2. Assign new initial positions.*

Initial positions
Default: random
Better starting points
Heuristics based on degree of nodes

A$er switching episode
1. Store old positions for existing objects.
2. Assign new initial positions.*
3. Run simulation without updating <svg> for n rounds
4. Animate objects from old to new positions.
5. Resume simulation and update <svg> every tick.

Animate Nodes & Links
Remove
delay
Move & Change size/thickness
Add new

const selection = svg.selectAll('g.node')
.data(nodes, d => d.entity.id);
selection.exit()
.transition()
.duration(1000)
.style('opacity', 0)
.remove();
const sEnter = selection.enter().append('g')
.classed('node', true)
.attr('transform', d => `translate(${d.x},${d.y})`)
.style('opacity', 0)
.call(force.drag);
sEnter.append('circle')
.attr('r', d=>d.r)
.style('fill', d => options.colorScale(d.entity.group));
const sTrans = selection.transition()
.delay(1000)
.duration(2000)
.attr('transform', d => `translate(${d.x},${d.y})`)
.style('opacity', 1)
sTrans.select('circle')
.attr('r', d=>d.r)
Add “enter” nodes
with opacity 0
After 1s delay,
use transition to move nodes
and fade in new nodes
Fade “exit” nodes to opacity 0
and remove
Create selection

Animate Communities
Remove
delay
Move & Change shape*
Add new
http://blockbuilder.org/kristw/f9ffe87dd8b4038b5867e853c27cebb7

Default
t=0 t=1

Smoother
t=0 t=1t=0.5 t=0.51

Code
// original
path.attr('d', hull);
// with custom interpolation
path.attrTween('d', (d,i,currentAttr) =>
interpolateHull(d, currentAttr)
)

Colors
Default: d3.category10()
Distinct but nothing about the context
Custom palette
Colors related to the groups/houses.
Black = Night’s Watch
Blue = North
Red = Daenerys
Gold = Lannister
…

Hold the vis.
CHAPTER V

The vis is not enough.

Legend

Navigation

Top 3

Adjust threshold

Recap

Filtered Recap
Tooltip

Demo
https://interactive.twitter.com/game-of-thrones

Mobile Support

A visualizer always evaluates his work.
CHAPTER VI

“Feedback is the breakfast of champion.”
— Ken Blanchard

Self & Peer
Does it solve the problem?

Google Analytics
Pageviews
Visitors
Actions
Referrals
Sites/Social

Feedback

Feedback

You know nothing, algorithm.
CHAPTER VII

Pros
List of Characters + Tweets = Vis.
No story or relationship as input

Cons
Weak assumption about connections
Can misclassify
Noise

All talk must end.
CHAPTER VIII

Summary
Problem first, not solution backwards
Prototype, Iterate, Scale & Adapt
Identify characters from Tweets + Network visualization
Vis is important, but there are other parts.
Feedback
Understand the pros & cons
Rooms for improvement
kristw.yellowpigz.com
Krist Wongsuphasawat / @kristw

Robert Harris, Miguel Rios, Elaine Filadelfo
and many colleagues at Twitter;
Elijah Meeks for his network vis tutorial;
Mike Bostock for D3 and examples
Lastly, to my wife for taking care of our baby, so I had time to prepare these slides.
Acknowledgement

Resources
Quora: How would you explain the plot of Game of Thrones in Brief from the beginning?
https://www.quora.com/How-would-you-explain-the-plot-of-Game-of-Thrones-in-brief-from-the-beginning
Convex Hull
https://en.wikipedia.org/wiki/Convex_hull
Images
Aemon - http://tinyurl.com/zqdzc2a
Cersei - http://tinyurl.com/gumcs7g
Crow - http://tinyurl.com/ju6up5g
Eddard - http://tinyurl.com/jc23mel
Feast - http://tinyurl.com/zcms4lq
Hammer - http://tinyurl.com/hz8emrp
Hodor - http://tinyurl.com/j748jfr
House sigils - http://tinyurl.com/jywgcjx
House Stark - http://tinyurl.com/jtdtrdy
Jon Snow - http://tinyurl.com/h4hofe8
Tyrion - http://tinyurl.com/z7z2uow
Tyrion - http://tinyurl.com/hvw4u89
Tyrion - http://tinyurl.com/jkrvqtb
Watercolor Map by Stamen Design

Thank you

Questions?