The document discusses modeling the US Congress as a graph database using Neo4j and analyzing relationships between legislators to identify influential members. It describes loading congressional data into Neo4j, querying relationships between legislators and states they represent. Methods for identifying influential legislators include degree centrality, betweenness centrality, and PageRank computed on a bill co-sponsorship graph using both Neo4j and Apache Spark with GraphX.

1. Congressional PageRank:
Graph Analytics Of US Congress
William Lyon
Graph Day - Austin, TX
January 2016

2. About me
Software Developer @Neo4j
william.lyon@neo4j.com
@lyonwj
lyonwj.com
William Lyon

3. Agenda
• Brief intro to Neo4j graph database
• Modeling US Congress as a graph
• Exploring the 114th Congress
• Finding influential legislators

4. Neo4j – Key Features
Native Graph Storage
Ensures data consistency and
performance
Native Graph Processing
Millions of hops per second, in real time
“Whiteboard Friendly” Data Modeling
Model data as it naturally occurs
High Data Integrity
Fully ACID transactions
Powerful, Expressive Query
Language
Requires 10x to 100x less code than
SQL
Scalability and High Availability
Vertical and horizontal scaling
optimized for graphs
Built-in ETL
Seamless import from other databases
Integration
Drivers and APIs for popular languages
MATCH
(A)

5. Property Graph Model

6. The Whiteboard Model Is the Physical Model

7. Relational Versus Graph Models
Relational Model Graph Model
KNOWS
KNOWS
KNOWS
ANDREAS
TOBIAS
MICA
DELIA
Person FriendPerson-Friend
ANDREAS
DELIA
TOBIAS
MICA

8. Property Graph Model Components
Nodes
• The objects in the graph
• Can have name-value properties
• Can be labeled
Relationships
• Relate nodes by type and
direction
• Can have name-value properties
CAR
DRIVES
name: “Dan”
born: May 29, 1970
twitter: “@dan”
name: “Ann”
born: Dec 5, 1975
since:
Jan 10, 2011
brand: “Volvo”
model: “V70”
LOVES
LOVES
LIVES WITH
OW
NS
PERSON PERSON

9. Cypher Query Language

10. Cypher: Powerful and Expressive Query
Language
CREATE (:Person { name:“Dan”} ) -[:LOVES]-> (:Person { name:“Ann”} )
LOVES
Dan Ann
LABEL PROPERTY
NODE NODE
LABEL PROPERTY

11. MATCH (boss)-[:MANAGES*0..3]->(sub),
(sub)-[:MANAGES*1..3]->(report)
WHERE boss.name = “John Doe”
RETURN sub.name AS Subordinate,
count(report) AS Total
Express Complex Queries Easily with Cypher
Find all direct reports and how
many people they manage,
up to 3 levels down
Cypher Query
SQL Query

12. http://www.opencypher.org/

13. Getting Data into Neo4j
Cypher-Based “LOAD CSV” Capability
• Transactional (ACID) writes
• Initial and incremental loads of up to
10 million nodes and relationships
Command-Line Bulk Loader
neo4j-import
• For initial database population
• For loads with 10B+ records
• Up to 1M records per second
4.58 million things
and their relationships…
Loads in 100 seconds!

14. Neo4j
Graph Database
• Property graph datamodel
• Nodes and relationships
• Native graph processing
• Cypher query language

15. Graphing US Congress

16. https://github.com/legis-graph/legis-graph

17. https://github.com/legis-graph/legis-graph
LOAD CSV WITH HEADERS
FROM “file:///legislators.csv” AS line
MERGE (l:Legislator (thomasID: line.thomasID})
SET l = line
MERGE (s:State {code:line.state})<-[:REPRESENTS]-(l)
…
US Congress

18. https://github.com/legis-graph/legis-graph

19. What Legislators represent Texas?
MATCH (s:State {code: "TX"})<-[:REPRESENTS]-(l:Legislator)
RETURN l,s;

20. …include congressional body and party
MATCH (s:State {code: "TX"})<-[:REPRESENTS]-(l:Legislator)
MATCH (p:Party)<-[:IS_MEMBER_OF]-(l)-[:ELECTED_TO]->(b:Body)
RETURN b,l,s,p;

24. How to find influential legislators?

25. Bill Sponsorship

27. Bill Cosponsorship

28. Degree centrality

29. Bill Cosponsorship

30. • Cosponsors are
“influenced by” bill
sponsors
• Add INFLUENCED_BY
relationships

31. Betweenness centrality
The number of times a node acts as a bridge
along the shortest path between two other nodes.
https://en.wikipedia.org/wiki/Betweenness_centrality

33. image credit: https://en.wikipedia.org/wiki/PageRank

34. image credit: https://en.wikipedia.org/wiki/PageRank
?

35. PageRank
Cypher approximation
UNWIND range(1,10) AS round
MATCH (l:Legislator)
WHERE rand() < 0.1
MATCH (l:Legislator)-[:INFLUENCED_BY]->(o:Legislator)
SET o.rank = coalesce(o.rank,0) + 1;
http://neo4j.com/blog/using-neo4j-hr-analytics/

36. Neo4j server extensions with Java

37. Neo4j server extensions with Java
curl http://localhost:7474/service/v1/pagerank/Person/KNOWS

38. PageRank
Graph processing server extension
https://github.com/maxdemarzi/graph_processing
curl http://localhost:7474/service/v1/pagerank/Person/KNOWS

39. PageRank
neo4j-noderank
https://github.com/graphaware/neo4j-noderank

40. Two issues
• Local vs global
• Iterative algorithms and graph complexity

41. Local vs global
Local Global

42. Local vs global
Local Global
Offline / batchOLTP / realtime

43. For iterative algorithms like PageRank, it’s all about complexity of the graph
Lots of paths. Lots of iterations
Graph complexity

44. PageRank
Graph global!

45. PageRank
Graph global!
Iterative!

46. • Efficient in-memory data processing and
machine learning platform
• Graph analytics with GraphX
• In-memory message passing algorithm
Apache Spark is a fast and general engine for large-scale data processing.
http://spark.apache.org/

48. PageRank
Spark with Neo4j - Scala
https://github.com/AnormCypher/AnormCypher

49. import org.anormcypher._
import org.apache.spark.graphx._
import org.apache.spark.graphx.lib._
val total = 100000000
val batch = total/1000000
val links = sc.range(0,batch).repartition(batch).mapPartitionsWithIndex( (i,p) => {
val dbConn = Neo4jREST("localhost", 9474, "/db/data/", "neo4j", "test")
val q = "MATCH (l1:Legislator)-[:INFLUENCED_BY]->(l2:Legislator) RETURN id(l1)
as from, id(l2) as to skip {skip} limit 1000000"
p.flatMap( skip => {
Cypher(q).on("skip"->skip*1000000).apply()(dbConn).map(row =>
(row[Int]("from").toLong,row[Int]("to").toLong)
)
})
})
links.cache
links.count
val edges = links.map( l => Edge(l._1,l._2, None))
val g = Graph.fromEdges(edges,"none")
val v = PageRank.run(g, 5).vertices
Extract subgraph. Run PageRank using Spark GraphX.

50. val res = v.repartition(total/100000).mapPartitions( part => {
val localConn = Neo4jREST("localhost", 9474, "/db/data/", "neo4j", "test")
val updateStmt = Cypher("UNWIND {updates} as update MATCH (p) where id(p) =
update.id SET p.pagerank = update.rank")
val updates = part.map( v => Map("id"->v._1.toLong, "rank" -> v._2.toDouble))
val count = updateStmt.on("updates"->updates).execute()(localConn)
Iterator(part.size)
})
Write back to graph

51. PageRank
Mazerunner
http://www.kennybastani.com/2014/11/using-apache-spark-and-neo4j-for-big.html
• Enables two-way ETL between
Spark and Neo4j
• Run GraphX jobs from data in
Neo4j
• Write results back to Neo4j

52. PageRank
Mazerunner
http://www.kennybastani.com/2014/11/using-apache-spark-and-neo4j-for-big.html
• Enables two-way ETL between
Spark and Neo4j
• Run GraphX jobs from data in
Neo4j
• Write results back to Neo4j
• Support for:
• PageRank
• Closeness Centrality
• Betweenness Centrality
• Triangle Counting
• Connected Components
• Strongly Connected Components

53. https://github.com/neo4j-contrib/neo4j-mazerunner

54. curl http://localhost:7474/service/mazerunner/analysis/pagerank/INFLUENCED_BY

55. • Cosponsors are
“influenced by” bill
sponsors
• Add INFLUENCED_BY
relationships

58. Who are the influential legislators?

59. Who are the influential legislators?

60. Influential legislators by topic

61. Influential legislators by topic

62. graphdatabases.com

63. http://graphgist.neo4j.com/

64. http://portal.graphgist.org/challenge/index.html

65. Links
• http://www.lyonwj.com/2015/09/20/legis-graph-congressional-data-
using-neo4j/
• http://www.lyonwj.com/2015/10/11/congressional-pagerank/
• https://github.com/legis-graph/legis-graph
• https://github.com/neo4j-contrib/neo4j-mazerunner
• http://www.kennybastani.com/2014/11/graph-analytics-docker-
spark-neo4j.html
• http://www.kennybastani.com/2015/03/spark-neo4j-tutorial-
docker.html