This document summarizes an analysis of complex networks using open source software tools. It provides an overview of network graph analysis and statistical and visual measures used to assess network patterns. It then demonstrates these concepts through case studies of the Miles Davis album collaboration network, the Boston Red Sox player network, and a GDELT news event network. The document concludes that network graph analysis is a powerful technique for understanding relationships in connected data.

1. Analyzing Complex Networks
Using Open Source Software
@ODSC
OPEN
DATA
SCIENCE
CONFERENCE
Ken Cherven
@kc2519
visual-baseball.com
visualidity.com
Boston | May 20-22nd 2016

2. A Brief Outline
• Network Graph Analysis overview
• Tools
• Case Studies
• Conclusions

3. Network Graph Analysis – aka Social
Network Analysis (SNA), is the study of
connections (links) between actors
(nodes) within a network
node
node
node
node
node

4. Network Graph Analysis has many use
cases, ranging from the familiar SNA
(Facebook, Twitter networks) to the
more specialized visual and statistical
investigation of political, criminal, or
terrorist networks

5. The use cases for Network Graph
Analysis are almost endless – any
dataset where relationships can be
mapped can be analyzed both
statistically and visually; all we need are
nodes and links

6. We have two primary approaches to
assess patterns in a network:
• Statistical measures are used to
understand the underlying structure and
relationships between nodes
• Visual assessment allows us to leverage
size, color, spacing, and structure to
understand patterns at a network level

7. Statistical measures are employed to
understand structural patterns within the
network:
• Degrees (# of connections)
• Centrality (influence)
• Density (level of network connectedness)
• Homophily (common groupings)
• Diameter (max distance between nodes)

8. Visual assessment allows us to use our
visual sense to interpret network
patterns:
• Node location to represent related nodes
• Node sizes to represent degrees
• Node coloring to represent common
groupings (clusters, categories)
• Edge weights that show the strength of
connections between nodes

9. Some open source network graph tools:
• Gephi (http://gephi.org)
• Cytoscape (http://cytoscape.org)
• GraphViz (http://graphviz.org)
• Sigma.js (http://sigmajs.org)
• NodeXL (http://nodexl.codeplex.com/)
• Pajek (http://mrvar.fdv.uni-lj.si/pajek/)
• Tulip (http://tulip.labri.fr/TulipDrupal/)

10. We’ll use Gephi and Sigma.js for the
following examples:
• Miles Davis album network (tripartite
network)
• Boston Red Sox player network
• GDELT event networks

11. Miles Davis Album Network

12. The desire behind the Miles Davis
network is to understand the multiple
phases within his long and varied
career, and to see the shifting
patterns in his musical partnerships
and styles
http://visual-baseball.com/gephi/jazz/miles_davis/#

13. Miles Davis Network Topology
Miles Davis
Albums
(pink)
Musicians
(colored by instrument)

14. Five Album Clusters to Investigate
2
3
1
4
5
What do these
clusters represent?

15. Five Album Clusters Revealed
Early 60s
Big
Bands
Mid-
60s
small
group
1950s
small
groups
1970s
fusion,
electric
sounds
Late career – 1980s,
experimentation, eclectic
instrumentation

16. A quick exploration of the network
reveals information about the elements
of time, instrumentation, number of
musicians, and types of instruments.
With just a few minutes of traversing the
network, we gain a greater
understanding of Miles Davis’ musical
career

17. Red Sox Historical Player Network

18. The goal for the Red Sox player
network is to understand connections
between players across eras, and to
understand influence and groupings
within the network, as defined by
degrees and other centrality
measures
http://visual-baseball.com/gephi/teams/redsox_network/

19. Red Sox Network Topology
Player nodes are sized and
colored based on number
of years with team and
cluster assignment
Players are positioned based
on common years with team
Links are built using the number
of seasons two players were
on the team roster together

20. Individual Network Footprints
19 Seasons
269 Degrees
6 Eccentricity
126,355 Betweenness
3.30 Closeness
Ted Williams

21. Individual Network Footprints
23 Seasons
283 Degrees
5 Eccentricity
596,003 Betweenness
2.64 Closeness
Carl Yastrzemski

22. Individual Network Footprints
15 Seasons
379 Degrees
7 Eccentricity
120,696 Betweenness
3.36 Closeness
Jason Varitek

23. A simple look at 3 prominent players
showed us some quickly observable
differences using centrality measures:
• Despite playing several fewer seasons
than either Williams or Yastrzemski,
Varitek has the most connections; but
Yastrzemski could get you to more
players faster by being very central to
the network structure

24. GDELT Network Analysis

25. GDELT data exposes an incredible
number of opportunities for viewing
network data based on published
accounts of news events around the
world. Our exploration focuses on US
Government threats reported
between March 1st and April 30, 2016

26. GDELT Network Topology (Geo Layout)
Using Geo Layout
Connections are between Actor1
and Actor2 within a specific event
instance; Actor1 is often the
Protagonist, Actor2 the Target
Nodes are positioned by lat/lon coordinates;
most are concentrated in the Northeast US
Node and edge colors are based on the
GDELT GoldsteinScale variable; darker colors
are indicative of higher destabilization potential

27. Exploring the Graph Geographically
Using Geo Layout

28. GDELT Network Topology (Dual Circle)
Using Dual Circle Layout
Prominent nodes are positioned in the inner
circle, based on the number of articles on
cumulative events (speeches, press
conference, negotiations, etc.)
Secondary nodes are positioned around the
outer circle; these may be either primary or
secondary actors in an event
Node colors are again based on the GDELT
GoldsteinScale variable

29. Exploring Nodes Using Sigma.js
Using Dual Circle Layout

30. Exploring Nodes Using Sigma.js
Using Dual Circle Layout

31. A few minutes of network exploration
reveals topic patterns based on news
reporting, and allows us to understand
which actors are directing actions
against others, and what is the tone of
those actions. Tracking these measures
over time will enable us to spot trends
both positive and negative.

32. Conclusions
• Network graph analysis is a powerful tool for
visually and statistically assessing complex
networks
• Network graphs are proliferating, due to the
availability of multiple open source tools and
increasing amounts of open data
• Network graph analysis can be used to tell
powerful stories wherever connected data is
present

33. Thanks –
and happy networking!

34. Backup

35. Miles Davis network specs:
• Data sourced from Wikipedia
• Nodes and edges created in Excel
• Graph created in Gephi using the Yifan Hu
Proportional algorithm
• Exported to Sigma.js (json format)
• 348 nodes, 596 edges

36. Red Sox Player Network specs:
• Data sourced from Lahman Database at
seanlahman.com
• Nodes and edges created using SQL code in
Toad for MySQL
• Graphs created in Gephi using the ARF layout
algorithm
• JSON file exported to Sigma.js
• 1668 nodes, 51,223 edges

37. GDELT classifications:
• Type refers to groupings such as
Government, Media, Education, and many
more
• Event codes reference the type of event –
riots, protests, sanctions, and so on
• The GoldsteinScale runs from -10 to 10 in
describing the relative destabilizing potential
of the event

38. GDELT Network specs:
• Data sourced from the GDELT event database
at gdeltproject.org (3/1 to 4/30/16)
• Nodes and edges refined using SQL code in
Toad for MySQL
• Graphs created in Gephi using the Geo Layout
and Dual Circle algorithms
• GEXF files exported for use with Sigma.js
• 414 nodes, 11,975 edges