GraphX is Apache Spark's API for graph distributed computing based on the Pregel programming model. In this talk we'll see a brief introduction to Pregel and then we'll focus on transforming standard graph algorithms in their distributed counterpart using GraphX to speedup performance in a distributed environment.

1. ThePregelProgrammingModelwith
SparkGraphX

2. Agenda
-GraphXIntroduction
-Pregelprogrammingmodel
-Codeexamples
Themainfocuswillbeontheprogrammingmodel

3. GraphXisagraphprocessing
systembuiltontopofApacheSpark
-propertygraphrepresentation
-basedonRDDs
-userdefinedpartitioningonRDDs

4. GraphX/Sparksoftwarestack

5. PregelProgrammingModel
https://kowshik.github.io/JPregel/pregel_paper.pdf
-basedonvertices
-messagesfrom/toneighbours
-boundedinsupersteps
-status(active/inactive)

6. PregelSample:findingthemaximumvalue

7. GraphXimplementationofPregel
Usesthreefunctions:
-vprogcomputesthenewvertexvalue
-sendMsgdecidestowhomsendthenewvalue
-mergeMsgmergesincomingvalues

8. GraphXcommunicationdiagram

9. graph.pregel(
initialMsg = Int.MinValue,
maxIterations = Int.MaxValue,
activeDirection = EdgeDirection.Out
)(
// vprog
(vertexId: Long, currentVertexAttr: Int, newVertexAttr: Int) =>
if (newVertexAttr > currentVertexAttr)
newVertexAttr else currentVertexAttr,
// sendMsg
(edgeTriplet: EdgeTriplet[Int, Int]) => {
if (edgeTriplet.srcAttr > edgeTriplet.dstAttr)
Iterator( (edgeTriplet.dstId, edgeTriplet.srcAttr) )
else
Iterator.empty
},
// mergeMsg
(attribute1: Int, attribute2: Int) =>
if (attribute1 > attribute2) attribute1 else attribute2
)
MaxValueimplementation

10. Graph initial state
Node [1]: 3
Node [2]: 6
Node [3]: 2
Node [4]: 1
Graph final state
Node [1]: 6
Node [2]: 6
Node [3]: 6
Node [4]: 6
Max value of the graph is 6.
MaxValueimplementation
Results:

11. Dijkstra'salgorithm
Unvisitednodes:
-Baltimore
-Detroit
-Chicago
-NewYork
-Philadelphia

12. Dijkstra'salgorithm
Unvisitednodes:
-Baltimore
-Detroit
-Chicago
-NewYork
-Philadelphia

13. Dijkstra'salgorithm
Unvisitednodes:
-Baltimore
-Detroit
-Chicago
-NewYork
-Philadelphia

14. Dijkstra'salgorithm
Unvisitednodes:
-Baltimore
-Detroit
-Chicago
-NewYork
-Philadelphia

15. Dijkstra'salgorithm
Unvisitednodes:
-Detroit
-Chicago
-NewYork
-Philadelphia

16. Dijkstra'salgorithm
Unvisitednodes:
-Detroit
-Chicago
-NewYork
-Philadelphia

17. Dijkstra'salgorithm
Unvisitednodes:
-Detroit
-Chicago
-NewYork
-Philadelphia

18. Dijkstra'salgorithm
Unvisitednodes:
-Chicago
-NewYork
-Philadelphia

19. Dijkstra'salgorithm
Unvisitednodes:
-Chicago
-NewYork
-Philadelphia

20. Dijkstra'salgorithm
Unvisitednodes:
-Chicago
-Philadelphia

21. Dijkstra'salgorithm
Unvisitednodes:
-Chicago
-Philadelphia

22. Dijkstra'salgorithm
Unvisitednodes:
-Chicago

23. Dijkstra'salgorithm
Unvisitednodes:

24. type VertexId = scala.Long
case class City(
name: String,
id: VertexId
)
case class VertexAttribute(
cityName: String,
distance: Double,
path: List[City]
)
Dijkstra'salgorithmimplementation
Typesdefinitions:

25. val shortestPathGraph = initialGraph.pregel(
initialMsg = VertexAttribute(
"",
Double.PositiveInfinity,
List[City]()
),
maxIterations = Int.MaxValue,
activeDirection = EdgeDirection.Out
)(
vprog,
sendMsg,
mergeMsg
)
Dijkstra'salgorithmimplementation

26. val vprog = (
vertexId: VertexId,
currentVertexAttr: VertexAttribute,
newVertexAttr: VertexAttribute
) =>
if (currentVertexAttr.distance <= newVertexAttr.distance) {
currentVertexAttr else newVertexAttr
}
val mergeMsg = (
attribute1: VertexAttribute,
attribute2: VertexAttribute
) =>
if (attribute1.distance < attribute2.distance) {
attribute1 else attribute2
}
Dijkstra'salgorithmimplementation

27. val sendMsg = (edgeTriplet: EdgeTriplet[VertexAttribute, Double]) =>
{
if (edgeTriplet.srcAttr.distance < (edgeTriplet.dstAttr.distance - edgeTriplet.attr)) {
Iterator( (
edgeTriplet.dstId,
new VertexAttribute(
edgeTriplet.dstAttr.cityName,
edgeTriplet.srcAttr.distance + edgeTriplet.attr,
edgeTriplet.srcAttr.path :+ new City(
edgeTriplet.dstAttr.cityName,
edgeTriplet.dstId
)
)
)
)
}
else Iterator.empty
}
Dijkstra'salgorithmimplementation

28. Going from Washington to Chicago has a distance of 105.0 km.
Path is: Washington [1] => Baltimore [2] => Detroit [3] => NewYork [5] => Chicago [4]
Going from Washington to Washington has a distance of 0.0 km.
Path is: Washington [1]
Going from Washington to Philadelphia has a distance of 91.0 km.
Path is: Washington [1] => Baltimore[2] => Detroit[3] => NewYork[5] => Philadelphia[6]
Going from Washington to Detroit has a distance of 62.0 km.
Path is: Washington [1] => Baltimore [2] => Detroit [3]
Going from Washington to NewYork has a distance of 76.0 km.
Path is: Washington [1] => Baltimore [2] => Detroit [3] => NewYork [5]
Going from Washington to Baltimore has a distance of 27.0 km.
Path is: Washington [1] => Baltimore [2]
Dijkstra'salgorithmimplementation
Results:

29. Questions&Answers

30. Thanks!
Thecodeisavailableat
https://github.com/andreaiacono/TalkGraphX