The document discusses Spark GraphX and Pregel algorithms for graph processing. It introduces GraphX and how it represents graphs as RDDs. It then covers the feedback vertex set algorithm for finding cycles in a graph and how it can be parallelized in GraphX. Finally, it discusses Pregel and how it allows for large-scale distributed graph computations through message passing between vertices.

1. Spark GraphX & Pregel
Challenges and Best Practices
Ashutosh Trivedi (IIIT Bangalore)
Kaushik Ranjan (IIIT Bangalore)
Sigmoid-Meetup Bangalore
https://github.com/anantasty/SparkAlgorithms

2. Agenda
• Introduction to GraphX
– How to describe a graph
– RDDs to store Graph
– Algorithms available
• Application in graph algorithms
– Feedback Vertex Set of a Graph
– Identifying parallel parts of the solution.
• Challenges we faced
• Best practices
Ashutosh & Kaushik, Sigmoid-Meetup
Bangalore Dec-2014
2

3. Graph Representation
Ashutosh & Kaushik, Sigmoid-Meetup
Bangalore Dec-2014
3
class Graph [ V, E ] {
def Graph(vertices: Table[ (Id, V) ],
edges: Table[ (Id, Id, E) ])
• The VertexRDD[A] extends RDD[(VertexID, A)] and adds the additional
constraint that each VertexID occurs only once.
• Moreover, VertexRDD[A] represents a set of vertices each with an
attribute of type A
• The EdgeRDD[ED], extends RDD[Edge[ED]]

4. Ashutosh & Kaushik, Sigmoid-Meetup
Bangalore Dec-2014
4
GraphX - Representation

5. Ashutosh & Kaushik, Sigmoid-Meetup
Bangalore Dec-2014
5
GraphX - Representation

6. Ashutosh & Kaushik, Sigmoid-Meetup
Bangalore Dec-2014
6
A BA
Vertex and Edges
Vertex Edge

7. Triplets Join Vertices and Edges
• The triplets operator joins vertices and edges:
TripletsVertices
B
A
C
D
Edges
A B
A C
B C
C D
A BA
B A C
B C
C D
Ashutosh & Kaushik, Sigmoid-Meetup
Bangalore Dec-2014
7

8. Ashutosh & Kaushik, Sigmoid-Meetup
Bangalore Dec-2014
8
Triplets elements

9. Ashutosh & Kaushik, Sigmoid-Meetup
Bangalore Dec-2014
9
Subgraphs
Predicates vpred and epred

10. Ashutosh & Kaushik, Sigmoid-Meetup
Bangalore Dec-2014
10
Feedback Vertex Set
• A feedback vertex set of a graph is a set of vertices whose removal
leaves a graph without cycles.
• Each feedback vertex set contains at least one vertex of any cycle in the
graph.
• The feedback vertex set problem is an NP-complete problem
in computational complexity theory
• Enumerate each simple cycle.

11. Ashutosh & Kaushik, Sigmoid-Meetup
Bangalore Dec-2014
11
1 2
34
5
6
7
8
9
10
Strongly Connected Components
Each strongly connected component can be considered in
parallel since they do not share any cycle
SC1 – (1) SC2 – (5) SC3 – (8) SC4 – (9)

12. Ashutosh & Kaushik, Sigmoid-Meetup
Bangalore Dec-2014
12
FVS Algorithm
#Greedy recursive solution
FVS(G)
sccGraph = scc(G)
For each graph in sccGraph
For each vertex
remove vertex and again calculate scc,
vertexV = vertex which give max number of scc #which means it
kills maximum cycles
subGraph = subgraph(removeV )
FVS (subGraph )

13. Ashutosh & Kaushik, Sigmoid-Meetup
Bangalore Dec-2014
13
1 2
4 3
2
4 3
Graph Iteration SCC count
3
1
4 3
1
1 2
4
3
1 2
4 3
1 2
4 3
Remove 2
Remove 1
Remove 3

14. Ashutosh & Kaushik, Sigmoid-Meetup
Bangalore Dec-2014
14
1
5
8 9
1 5 8 9Feedback Vertex Set

15. Ashutosh & Kaushik, Sigmoid-Meetup
Bangalore Dec-2014
15
FVS – Spark Implementation
sccGraph has one more property sccID on each vertices, extract it

16. Ashutosh & Kaushik, Sigmoid-Meetup
Bangalore Dec-2014
16
FVS – Spark Implementation
sccGraph = scc(G)
For each graph in sccGraph

17. Ashutosh & Kaushik, Sigmoid-Meetup
Bangalore Dec-2014
17
FVS – Spark Implementation
#Greedy recursive function

18. Ashutosh & Kaushik, Sigmoid-Meetup
Bangalore Dec-2014
18
FVS – Spark Implementation
For each vertex
remove vertex and again calculate scc,
# Z is a list of scc count after removing each vertex

19. Ashutosh & Kaushik, Sigmoid-Meetup
Bangalore Dec-2014
19
vertexV = vertex which give max number of scc #which means it
kills maximum cycles
FVS – Spark Implementation

20. Ashutosh & Kaushik, Sigmoid-Meetup
Bangalore Dec-2014
20
subGraph = subgraph(removeV )
FVS (subGraph )
FVS – Spark Implementation

21. Ashutosh & Kaushik, Sigmoid-Meetup
Bangalore Dec-2014
21
Pregel
• Graph DB
– Data Storage
– Data Mining
• Advantages
– Large-scale distributed computations
– Parallel-algorithms for graphs on multiple machines
– Fault tolerance and distributability

22. Ashutosh & Kaushik, Sigmoid-Meetup
Bangalore Dec-2014
22
Oldest Follower
What is the age of oldest follower of each user ?
Val oldestFollowerAge = graph
.aggregateMessages(
#map word => (word.dst.id, word.src.age),
#reduce (a,b) => max(a, b)
)
.vertices
mapReduceTriplets is now aggregateMessages

23. Ashutosh & Kaushik, Sigmoid-Meetup
Bangalore Dec-2014
23
In aggregateMessages :
• EdgeContext which exposes the triplet fields .
• functions to explicitly send messages to the source and
destination vertex.
• It require the user to indicate what fields in the triplet are
actually required.
New in GraphX

24. Theory – it’s Good
How it works – that’s awesome
Ashutosh & Kaushik, Sigmoid-Meetup
Bangalore Dec-2014
24
Graph’s are recursive data-structures, where the
property of a vertex is dependent on the properties of
it’s neighbors, which in turn are dependent on the
properties of their neighbors.

25. Graph.Pregel ( initialMessage ) (
#message consumption
( vertexID, initialProperty, message ) → compute new property
,
#message generation
triplet → .. code ..
Iterator( vertexID, message )
Iterator.empty
,
#message aggregation
( existing message set, new message ) → NEW message set
)
Ashutosh & Kaushik, Sigmoid-Meetup
Bangalore Dec-2014
25
Architecture

26. Ashutosh & Kaushik, Sigmoid-Meetup
Bangalore Dec-2014
26
1 2
4 3
1030
30 20
1 2
4 3
10
30
30 20
max [30,10,20]
max [20] max [10]
1 2
4 3
100
10 10
1 2
4 3
10
0
10 10
max [10] max [10]

27. Ashutosh & Kaushik, Sigmoid-Meetup
Bangalore Dec-2014
27
Example - output
1 2
4 3
100
0 0

28. Applications - GIS
• Algorithm – to compute all vertices in a directed graph, that can
reach out to a given vertex.
• Can be used for watershed delineation in Geographic Information
Systems
Ashutosh & Kaushik, Sigmoid-Meetup
Bangalore Dec-2014
28
Vertices that can reach out to E are A and B

29. Algorithm
Graph.Pregel( Seq[vertexID’s] ) (
#message consumption
if vertex.state == 1
vertex.state → 2
else if vertex.state == 0
if ( vertex.adjacentVertices ∩ Seq[ vertexID’s ] ) isNotEmpty
vertex.state → 2
#message aggregator
Seq[existing vertex ID’s] U Seq[new vertex ID]
)
Ashutosh & Kaushik, Sigmoid-Meetup
Bangalore Dec-2014
29

30. Ashutosh & Kaushik, Sigmoid-Meetup
Bangalore Dec-2014
30
#message generation
for each triplet
if destinationVertex.state == 1
message( sourceVertexID, Seq[destinationVertexID] )
message( destinationVertexID, Seq[destinationVertexID] )
else if sourceVertex.state == 1 and destinationVertex.state == 2
message( sourceVertexID, Seq[destinationVertexID] )
else message( empty )
Algorithm

31. References
• Fork our repository at
• https://github.com/anantasty/SparkAlgorithms
• Follow us at
• https://github.com/codeAshu
• https://github.com/kaushikranjan
• https://spark.apache.org/docs/latest/graphx-programming-guide.html
Ashutosh & Kaushik, Sigmoid-Meetup
Bangalore Dec-2014
31