Pregel and giraph

1. Pregel

2. Pregel
• A System for Large-Scale Graph Processing
• Sufficiently flexible to express arbitrary graph
algorithms
• So easy

3. Pregel: Model Of Computation
• Vertex state
• Terminate codition: all vertex are inactive

4. Pregel: Model Of Computation
• Sequence of supersteps
• Invoke compute() for each active vertex
• Each vertex can
– Modify its state, its outgoing edges
– Recive messages
– Send messages to another

5. Pregel: Model Of Computation

6. Pregel: Model Of Computation

7. Pregel API

8. Pregel API
• Combiners
• Aggregators
• Topology Mutations
• Input and Output

9. Giraph

10. Why not implement Giraph with
multiple MapReduce jobs
• Too much disk, no in-memory caching, a
superstep becomes a job!

11. Giraph is a single Map-only job in
Hadoop
• Hadoop is purely a resource manager for Giraph, all
communication is done through Netty-based IPC

12. Maximum vertex value
implementation

13. Giraph components
• Master
– One active master at a time
– Assign partition owners to workers prior to each
superstep
– Synchronize supersteps
• Worker
– Load the graph from input
– Does the computation/messaging of its assigned
partitions

14. Graph distribution