The document discusses large-scale graph processing and summarizes several graph processing tools and techniques. It describes Google's Pregel framework, which introduced the bulk synchronous parallel computation model. It also discusses Apache Giraph, an open-source implementation of Pregel, and how it uses Hadoop and ZooKeeper. Finally, it summarizes more recent tools like Spark and GraphX that can perform graph analytics in-memory at faster speeds.

1. Thorny path to the
Large-Scale Graph
Processing
Zinoviev Alexey

2. About
• I am a <graph theory, machine learning, traffic jams prediction, BigData
algorythms> scientist
• But I'm a <Java, JavaScript, Android, NoSQL, Hadoop, Spark>
programmer

3. BigData & Graph Theory
3/65

4. Big Data of old times
• Astronomy
• Weather
• Trading
• Sea routes
• Battles

5. And now ...
• Web graph
• Facebook friend network
• Gmail email graph
• EU road network
• Citation graph
• PayPal transaction graph

6. Graph Number of
vertexes
Number of
edges
Volume Data/per day
Web-graph 1,5 * 10^12 1,2 * 10^13 100 PB 300 TB
Facebook
1,1 * 10^9 160 * 10^9 1 PB 15 TB
(friends
graph)
Road graph
of EU
18 * 10^6 42 * 10^6 20 GB 50 MB
Road graph
of this city
250 000 460 000 500 MB 100 KB

7. Problems
• Popularity rank (page rank)
• Determining popular users, news, jobs, etc.
• Shortest paths
• Max flow
• How are users, groups connected?
• Clustering, semi-clustering
• Max clique, triangle closure, label propagation algorithms
• Finding related people, groups, interests

12. Node Centrality Problem
• Verticies with high impact
• Removal of important vertices reduces the reliability
Cases:
• Bioinformatics
• Social connections
• Road network
• Spam detection
• Recommendation system

14. Small World Problem
Facebook 4.74 712 M 69 G
Twitter 3.67 ---- 5G follows
MSN Messenger
(1 month)
6.6 180 M 1.3 G arcs

15. Large graph processing tools
15/65

16. Think like a vertex…
• Majority of graph algorithms are iterative and traverse the graph in
some way
• Classic map-reduce overheads (job startup/shutdown, reloading data
from HDFS, shuffling)
• High complexity of graph problem reduction to key-value model
• Iteration algorythms, but multiple chained jobs in M/R with full saving
and reading of each state

17. Why not use MapReduce/Hadoop?
• Example: PageRank, Google‘s
famous algorithm for measuring the
authority of a webpage based on the
underlying network of hyperlinks
• defined recursively: each vertex
distributes its authority to its neighbors
in equal proportions

18. Google Pregel
• Distributed system especially developed for large scale graph
processing
• Bulk Synchronous Parallel (BSP) as execution model
• Supersteps are atomic units of parallel computation
• Any superstep can be restarted from a checkpoint (need not be user
defined)
• A new superstep provides an opportunity for rebalancing of
components among available resources

19. Superstep in BSP

20. Vertex-centric BSP
• Each vertex has an id, a value, a list of its adjacent vertex ids and the
corresponding edge values
• Each vertex is invoked in each superstep, can recompute its value and
send messages to other vertices, which are delivered over superstep
barriers
• Advanced features : termination votes, combiners, aggregators,
topology mutations

21. C++ API, Pregel

23. Apache Giraph
23/65

24. Why Apache Giraph
Pregel is proprietary, but:
• Apache Giraph is an open source implementation of Pregel
• Runs on standard Hadoop infrastructure
• Computation is executed in memory
• Can be a job in a pipeline(MapReduce, Hive)
• Uses Apache ZooKeeperfor synchronization

26. Why Apache Giraph
• No locks: message-based communication
• No semaphores: global synchronization
• Iteration isolation: massively parallelizable

29. ZooKeeper in Apache Giraph
ZooKeeper: responsible for
computation state
• Partition/worker mapping
• Global state: superstep
• Checkpoint paths, aggregator
values, statistics

30. Master in Apache Giraph
Master: responsible for coordination
• Assigns partitions to workers
• Coordinates synchronization
• Requests checkpoints
• Aggregates aggregator values
• Collects health statuses

31. Worker in Apache Giraph
Worker: responsible for vertices
• Invokes active vertices
compute() function
• Sends, receives and assigns
messages
• Computes local aggregation
values

32. Scaling Giraph to a trillion edges

33. Fault tolerance
No single point of failure from Giraph threads
• With multiple master threads, if the current master dies, a new
one will automatically take over.
• If a worker thread dies, the application is rolled back to a
previously checkpointed superstep.
• If a zookeeper server dies, as long as a quorum remains, the
application can proceed
Hadoop single points of failure still exist (Namenode, jobtracker)

34. Worker Scalability, 250m nodes

35. Vertex scalability, 300 workers

36. Vertex/workers scalability

37. MapReduce vs Giraph
6 machines with 2x8core Opteron CPUs, 4x1TB disks and 32GB RAM each, ran 1
Giraph worker per core
Wikipedia page link graph (6 million vertices, 200 million edges)
PageRank on Hadoop/Mahout
• 10 iterations approx. 29 minutes
• average time per iteration: approx. 3 minutes
PageRank on Giraph
• 30 iterations took approx. 15 minutes
• average time per iteration: approx. 30 seconds
10x performance improvement

38. Okapi
• Apache Mahout for graphs
• Graph-based recommenders: ALS,
SGD, SVD++, etc.
• Graph analytics: Graph
partitioning, Community Detection,
K-Core, etc.

39. Giraph’s killer

40. Spark
• MapReduce in memory
• Up to 50x faster than Hadoop
• Support for Shark (like Hive), MLlib
(Machine learning), GraphX (graph
processing)
• RDD is a basic building block
(immutable distributed collections of
objects)

41. Spark in Hadoop old family

42. GraphX
Supported algorythms
● PageRank
● Connected components
● Label propagation
● SVD++
● Strongly connected components
● Triangle count

43. GraphChi
• Asynchronous Disk-based version of GraphLab
• Utilizing parallel sliding window
• Very small number of non-sequential accessesto the disk
• Graph does not fit in memory
• Input graph is split into P disjoint intervals to balance edges,
each associated with a shard
• For Home deals ...

44. GraphChi

45. GraphChi

46. Road Networks
46/65

47. Definition
• Edge weights > 0
• A few classes of roads
• Lat/Lon attributes for each vertex
• Subgraphs for cross-roads
• Not so big as web graph
• Static

48. Shortest path problem

49. AI

50. Full

51. Dijkstra

52. Bi-Directional

53. We need in fast system!
• Response < 10 ms (with high accuracy)
• Shortest path (SP) with O(n)
• Preprocessing phase
• Don’t keep all SP - O(n^2)
• Use geo attributes
• Using compression and recoding for
disk storage
• Network is stable

54. EU Road network
Dijkstra ALT RE HH CH TN HL
2 008 300 24 656 2444 462.0 94.0 1.8 0.3
• ALT: [Goldberg & Harrelson 05], [Delling & Wagner 07]
• RE: [Gutman 05], [Goldberg et al. 07]
• HH: [Sanders & Schultes 06]
• CH: [Geisberger et al. 08]
• TN: [Geisberger et al. 08]
• HL: [Abraham et al. 11]

55. A* with landmarks (ALT)

56. Reach (RE)

57. Transit nodes (TN)
• Divide graph G on subgraphs G_i
• Find R (subset of G_i) for each G_i
• All sortest path in G_i across R
• Build pairs (v_i, r_k) for each v_i where
r_k is closest Transit Node
• Calculate shortest paths between transit
nodes in R
• Save it!

58. TN + ALT

59. Special Cases
59/65

60. Optimization problems
• Unstable graph
• Prerpocessing phase is meaningless
• How to invest 1B $ in road network to minimize human time in
traffic jams
• How to invest 1M $ in road network to improve reliability before
the flooding

61. Last steps ...
• I/O Efficient Algorythms and Data Structures
• Graphs and Memory Errors

62. Omsk

63. Novosibirsk

64. Novosibirsk, TN preprocessing

65. twitter + G+ + VK