Northwestern data visualization - why why why

Northwestern Data Visualization| @gabegaster | 2015 may
what is data science?

who is a data scientist?

review of literature

“a scientist who can code”

• lower barrier to attack new problems

• repeatable analysis

• repeatable analysis
• freedom to think about problems new ways

using emerging technologies to approach
problems scientiﬁcally

which were difﬁcult to answer before

computing has progressed

1950

1950
cost of new
analysis

1950
cost of new
analysis
years

1950
cost of new
analysis
years
today

1950
cost of new
analysis
years
today
v

1950
cost of new
analysis
years
today
hoursv
v

1950
cost of new
analysis
years
today
same person thinking about the problem
can conduct experiments to answer it
hoursv
v

open-source code

open-source code
standing on
shoulders of giants

open-source code
standing on
shoulders of giants
reinventing the wheel

knowing
what is possible

knowing
what is possible
doing
something useful

knowing
what is possible
doing
something useful
HOW

knowing
what is possible
doing
something useful
HOW WHY

knowing
what is possible
doing
something useful
using
new
good
the right
tools

knowing
what is possible
doing
something useful
using
new
good
the right
asking whytools

knowing
what is possible
doing
something useful
using
new
good
the right
asking why
tools

knowing
what is possible
doing
something useful
using
new
good
the right
asking whytools WHY

knowing
what is possible
doing
something useful
using
new
good
the right
asking whytools WHY
WHY

why why why

why why why
science is about asking why

why why why
science is about asking why
start there

an anecdote

an example
from the real world

• e
an example

• what
what data do you have?
how can we make a network?

goal: save money

goal: save money
task: ﬁnd needle in the haystack (without poking yourself)

aboutpatent
not
aboutpatent
goal: save money

aboutpatent
not
aboutpatent
turn over to plaintiff
don’t
adverse inference
goal: save money

aboutpatent
not
aboutpatent
don’t
adverse inference
give away trade secrets
goal: save money

don’t
goal: save money

goal: save money
prototype

goal: save money
prototype — design for lawyers

Sexier. Less nerdy. Tailored.
design for transparency

http://www.daegis.com/judicial-acceptance-of-tar/

another example
contests

another example

task:

classify schizophrenia w MRItask:

why?

why?
improve understanding of disease

why?
how?

why?
how? … outside contest purview

why? outside contest purview

kaggle

kaggle
getting data
&
making usable

kaggle
getting data
&
making usable
WHY

timeline of contest
Accuracy of Classiﬁcation

timeline of contest
AUC

what is AUC?
AUC

AUC
what is AUC? Area Under Curve

AUC
what curve?

AUC
what curve? Receiver Operating
Characteristic

balances:
AUC
Characteristic

balances:
True Positive Rate
False Positive Rate
AUC
Characteristic

AUC
what is AUC?
balances:
True Positive Rate
False Positive Rate
Area Under Curve
Characteristic

why?
AUC
what is AUC?
balances:
True Positive Rate
False Positive Rate
Area Under Curve
Characteristic

why?
…
AUC
what is AUC?
balances:
True Positive Rate
False Positive Rate
Area Under Curve
Characteristic

why?
…
upshot:
AUC
what is AUC?
balances:
True Positive Rate
False Positive Rate
Area Under Curve
Characteristic

why?
…
choice of metric matters a LOT
upshot:
in practice
AUC
what is AUC?
balances:
True Positive Rate
False Positive Rate
Area Under Curve
Characteristic

timeline of contest
AUC

timeline of contest
AUC
random guess

timeline of contest
AUC
random guess
basic SVM

timeline of contest
goal?
AUC
random guess
basic SVM

timeline of contest
goal: depends on why
AUC
random guess
basic SVM

random guess
basic SVM
timeline of contest
AUC

me
timeline of contest
AUC

me
timeline of contest
AUC
turned out to place 9th — because overﬁtting

me
timeline of contest
AUC
turned out to place 9th — because overﬁtting
very common problem

timeline of contest
worth it?
AUC

We need to reduce the costs of Service Requests.
They are too expensive.
!
!
!
!
Thousands of engineers around the world, 24-7 read
through emails and hardware log ﬁles to determine
the cause of failure of a server. This is an expensive
process. We've tried to automate it. We can now
automatically resolve 7% of new Service Requests.
But we want more. That's why we bought a few
super computers with TBs of memory.
client
an example
!
from the industrial internet

Why? Why do you need to set up a hadoop architecture
to do clustering? What will this help you achieve?
!
!
!
!
!
How do you handle Service Requests?
!
!
!
!
!
client

!
!
!
!
!
!
!
!
!
!
!
client

!
!
!
!
!
How do you handle Service Requests?
!
!
!
!
!
client

client
tools are not everything

client
but it is important to know 
the right tool for the job

client
don’t start w hadoop unless you have to.
!

client
don’t start w hadoop unless you have to.
!
probably you don’t have to.

client
How did you automate resolving Service Requests?
!
!
!
!
!
!
!
!
!
!
!

client
!
!
!
!
!
!
!
!
!
!
!
A group of senior engineers thought about different use
cases and came up with a list of conditions that, if any
are met, lead to predetermined solutions.

client
!
!
!
!
!
!
!
!
!
!
!
!
Took a year to create.
!

client
!
!
!
!
!
!
!
!
!
!
!
!
!
We’ve been keeping track of every solved request for
several years now.

client
!
!
!
!
!
!
!
!
!
!
!
!
!
We’ve been keeping track of every solved request for
several years now.
from sklearn import naive_bayes as nb!
nb.GaussianNB().fit(historical_requests,!
! ! ! ! ! ! historical_decisions)

client
This works really well! But we can’t use it.
!
!
!
!
!
!

client
!
!
!
!
!
!
Oh. Why is that?

client
!
!
!
!
!
!
Engineers don’t trust the predictions.
Oh. Why is that?

an example
just for fun

an example
just for fun
(a bit more depth this time)

a typical trip for me

Bus transit
times
= a LIE

Chicago is a grid city

Difﬁcult
Public
Transit on
the grid
=+
Diagonals

Difﬁcult
Public
Transit on
the grid
=+
Diagonals
2+ buses = FAIL

Adding bikes to
public transit
=
win

show how has divvy
changed where people
can go
viz Goal:

show how has divvy
changed where people
can go
show where people
actually go
viz Goal:

How to show where
people go from here?

one way is with an arrow
A B
How to show where
people go from here?

good for abstract relationships
one way is with an arrow

hard to decipher

lines between pts?
@ﬂowingdata

emphasizes trafﬁc
@ﬂowingdata
lines between pts?
(the lines superimpose)

emphasizes trafﬁc
@ﬂowingdata
lines between pts?
beautiful map

emphasizes trafﬁc
@ﬂowingdata
lines between pts?
beautiful map
— but not suited for this goal

lines between pts?

lines between pts?
how to represent stations?

lines between pts?
on a map

but how?
lines between pts?
on a map

can use gradient — to
show gradual differences
between stations
London transit map
@mySociety

@mbostock
or use natural
borders?
London transit map
@mySociety

what regions?

each point is related to the
closest station
what regions?

closest station
what regions?
—> Voronoi

closest station
what regions?
—> Voronoi
huh?

closest station
what regions?
—> Voronoi
huh?
http://alexbeutel.com/webgl/voronoi.html

closest station
what regions?
—> Voronoi
huh?
http://alexbeutel.com/webgl/voronoi.html
Find the closest station — that’s my region!

Czech beer

Czech beer
starbucks

@gabegaster | http://bit.ly/1pdP2Tb
recap:

recap:
stations are voronoi tiles

@gabegaster | http://bitly.com/bundles/gabegaster/1
recap:

recap:
bubbles

recap:
too many edges to show…

recap:
too many edges to show…
!
what about using color?

how to
use color?

how to
use color?
colors v colors

colors !
v!
colors
@mySociety
@mbostock

how to
use color?
• two color scale
colors v colors

how to
use color?
binned v gradient
colors v colors
• two color scale

• hard to read
• differences subtle
gradient

how to
use color?
• two colors not many
• binned not gradient
binned v gradient
colors v colors

how to
use color?
binned v gradient
colors v colors
binned

how to
use color?
• transparent empty bin
binned v gradient
colors v colors
binned

how to
use color?
• transparent empty bin
• iterate
binned v gradient
colors v colors
binned

• dispersion
(where
people
can / do
go)
with these choices,
the viz emphasizes:
• exploration

what can we learn from this?
urban vs lake front
commute vs party
vs train
there’s me, too.
hipster vs yuppie

Great! This looks great!
!

!
But it’s too much at once.

!
make it interactive

!
make it interactive
divvy.datasco.pe

new question

How are stations different?

who uses it

when is the station used
who uses it

how it used
who uses it

how it used
who uses it
use the time signature of a station

Time Signature of a station

Time Signature of a station
http://divvy.datasco.pe/multiline/

questions?

in conclusion

thanks!
@gabegaster

Northwestern data visualization - why why why

Recommended

Recommended

More Related Content

Viewers also liked

Viewers also liked (20)

Similar to Northwestern data visualization - why why why

Similar to Northwestern data visualization - why why why (20)

Recently uploaded

Recently uploaded (20)

Northwestern data visualization - why why why