Spark Summit, profile picture
Uploaded bySpark Summit
PDF, PPTX3,513 views

A Tale of Two Graph Frameworks on Spark: GraphFrames and Tinkerpop OLAP Artem Aliev and Russell Spitzer

The document compares two graph analytics frameworks, Tinkerpop and GraphFrames, used with Apache Spark, highlighting their distinct approaches. Tinkerpop is a flexible OLAP framework that operates on a node-centric model, while GraphFrames offers a relational structure and integrates closely with Spark's DataFrame API for optimization. The choice of framework depends on the complexity of traversals and the need for interoperability with external data sources.

Data & Analytics◦
Related topics:
Apache Spark•

Embed presentation

Download as PDF, PPTX
Artem Aliev and Russell Spitzer, DataStax A Tale of Two Graph Frameworks on Spark: 
 GraphFrames and Tinkerpop OLAP #EUeco3
#EUeco3 Pierrot and Harlequin • Artem • Graph Analytics Expert • Earth • Russell • Distributed Systems Enthusiast • Earth 2
Tinkerpop and GraphFrames provide Complimentary Approaches for Graph Analytics DataSet Catalyst GraphFrames 3#EUeco3
Graphs are Vertices and Edges 4 Vertices are things and edges represent their relations to one another #EUeco3
Graphs are Vertices and Edges 5 Registry: USS Enterprise (NCC-1701-C) Class: Ambassador Service: 2332[11] – 2344 (12 Years) Registry: USS Enterprise (NCC-1701-D) Class: Galaxy Service: 2363–2371 (8 Years) Registry: USS Enterprise (NCC-1701) Class: Constitution class[6] Service: 2245–2285 (40 Years) Registry: USS Enterprise (NCC-1701-A) Class: Enterprise class[8][9] Service: 2286–2293 (7 Years) #EUeco3
Graphs are Vertices and Edges 6 Registry: USS Enterprise (NCC-1701-C) Class: Ambassador Service: 2332[11] – 2344 (12 Years) Registry: USS Enterprise (NCC-1701-D) Class: Galaxy Service: 2363–2371 (8 Years) Registry: USS Enterprise (NCC-1701) Class: Constitution class[6] Service: 2245–2285 (40 Years) Registry: USS Enterprise (NCC-1701-A) Class: Enterprise class[8][9] Service: 2286–2293 (7 Years) Vertex Properties #EUeco3
Graphs are Vertices and Edges 7 Registry: USS Enterprise (NCC-1701-C) Class: Ambassador Service: 2332[11] – 2344 (12 Years) Registry: USS Enterprise (NCC-1701-D) Class: Galaxy Service: 2363–2371 (8 Years) Registry: USS Enterprise (NCC-1701) Class: Constitution class[6] Service: 2245–2285 (40 Years) Registry: USS Enterprise (NCC-1701-A) Class: Enterprise class[8][9] Service: 2286–2293 (7 Years) succeeded by succeeded by succeeded by #EUeco3
Graphs are Vertices and Edges 8 Registry: USS Enterprise (NCC-1701-C) Class: Ambassador Service: 2332[11] – 2344 (12 Years) Registry: USS Enterprise (NCC-1701-D) Class: Galaxy Service: 2363–2371 (8 Years) Registry: USS Enterprise (NCC-1701) Class: Constitution class[6] Service: 2245–2285 (40 Years) Registry: USS Enterprise (NCC-1701-A) Class: Enterprise class[8][9] Service: 2286–2293 (7 Years) Edge Edge Labelsucceeded by succeeded by succeeded by #EUeco3
Graphs are Vertices and Edges 9 Registry: USS Enterprise (NCC-1701-C) Class: Ambassador Service: 2332[11] – 2344 (12 Years) Registry: USS Enterprise (NCC-1701-D) Class: Galaxy Service: 2363–2371 (8 Years) Registry: USS Enterprise (NCC-1701) Class: Constitution class[6] Service: 2245–2285 (40 Years) Registry: USS Enterprise (NCC-1701-A) Class: Enterprise class[8][9] Service: 2286–2293 (7 Years) Ship Ship Ship Ship Vertex Label succeeded by succeeded by succeeded by #EUeco3
Graphs are Vertices and Edges 10 Registry: USS Enterprise (NCC-1701-C) Class: Ambassador Service: 2332[11] – 2344 (12 Years) Registry: USS Enterprise (NCC-1701-D) Class: Galaxy Service: 2363–2371 (8 Years) Registry: USS Enterprise (NCC-1701) Class: Constitution class Service: 2245–2285 (40 Years) Ship Ship Ship Ship Position: Captain
 Name: Kirk Position: Captain
 Name: Picard Crew Crew succeeded by succeeded by succeeded by #EUeco3
Graphs are Vertices and Edges 11 Registry: USS Enterprise (NCC-1701-C) Class: Ambassador Service: 2332[11] – 2344 (12 Years) Registry: USS Enterprise (NCC-1701-D) Class: Galaxy Service: 2363–2371 (8 Years) Registry: USS Enterprise (NCC-1701) Class: Constitution class Service: 2245–2285 (40 Years) Registry: USS Enterprise (NCC-1701-A) Class: Enterprise class Service: 2286–2293 (7 Years) Ship Ship Ship Ship Position: Captain
 Name: Kirk Position: Captain
 Name: Picard Crew Crew succeeded by succeeded by succeeded by served on served on served on served on #EUeco3
Graphs are Vertices and Edges 12 Registry: USS Enterprise (NCC-1701-C) Class: Ambassador Service: 2332[11] – 2344 (12 Years) Registry: USS Enterprise (NCC-1701-D) Class: Galaxy Service: 2363–2371 (8 Years) Registry: USS Enterprise (NCC-1701) Class: Constitution class Service: 2245–2285 (40 Years) Registry: USS Enterprise (NCC-1701-A) Class: Enterprise class Service: 2286–2293 (7 Years) Ship Ship Ship Ship Position: Captain
 Name: Kirk Position: Captain
 Name: Picard Crew Crew succeeded by succeeded by succeeded by served on served on served on served on But why do I want this? #EUeco3
Graphs let us ask questions about our data based on their relations 13 What Captain Served After Kirk? What Ship was two after the NCC-1701? #EUeco3
Traversals involve following paths through the Graph 14 Registry: USS Enterprise (NCC-1701-C) Class: Ambassador Service: 2332[11] – 2344 (12 Years) Registry: USS Enterprise (NCC-1701-D) Class: Galaxy Service: 2363–2371 (8 Years) Registry: USS Enterprise (NCC-1701) Class: Constitution class Service: 2245–2285 (40 Years) Registry: USS Enterprise (NCC-1701-A) Class: Enterprise class Service: 2286–2293 (7 Years) Ship Ship Ship Ship Position: Captain
 Name: Kirk Position: Captain
 Name: Picard Crew Crew succeeded by succeeded by succeeded by served on served on served on served on #EUeco3
What Captain was After Kirk? 15 Registry: USS Enterprise (NCC-1701-C) Class: Ambassador Service: 2332[11] – 2344 (12 Years) Registry: USS Enterprise (NCC-1701-A) Class: Enterprise class Service: 2286–2293 (7 Years) Ship Ship Position: Captain
 Name: Kirk Position: Captain
 Name: Picard Crew Crewsucceeded by served on served on #EUeco3
What Ship was two after the NCC-1701? 16 Registry: USS Enterprise (NCC-1701-C) Class: Ambassador Service: 2332[11] – 2344 (12 Years) Registry: USS Enterprise (NCC-1701) Class: Constitution class Service: 2245–2285 (40 Years) Registry: USS Enterprise (NCC-1701-A) Class: Enterprise class Service: 2286–2293 (7 Years) Ship Ship Ship succeeded by succeeded by #EUeco3
Tinkerpop is a Powerful and Flexible Graph Framework • Server, Language, Connectors • Graph Framework for 
 OLAP and OLTP • Node Centric Representations • Fluent API (Gremlin) • Fully Self Contained Framework 17#EUeco3
OLTP Examples 18#EUeco3 18
Movie Lens Example Schema 19 https://grouplens.org/datasets/movielens/ #EUeco3 19
20
#EUeco3 What happens when you have too much data? 21
#EUeco3 Tinkerpop Spark OLAP Mechanism • Instead of one traversal we traverse starting from all nodes simultaneously 22
Distribution Requires Partitioning 23 ? Big Data Independent Chunks of Data#EUeco3
#EUeco3 Vertex Stored in a PairRDD Id -> StarVertex(Edge and Property Information) 24 1 A C D Star Vertex: Adjacency list representation
 1: "A", "Kirk"
 A: "C", "Kirk"
 C: "D", "Picard"
 D: "Picard"
 Just Id 
 Of Connected 
 Vertex
#EUeco3 Vertex Program Runs Initializing Traverser for every Vertex 25 1 A C D SparkMemory - Accumulator - Used for GlobalState
#EUeco3 Then we cycle through a message Passing Algorithm 26 1 A C D 1 A C D 1 A C D SparkMemory - Accumulator - Used for GlobalState
#EUeco3 Then we cycle through a message Passing Algorithm 27 1 A C D 1 A C D 1 A C D SparkMemory - Accumulator - Used for GlobalState Passes messages from one Vertex to another with a join
#EUeco3 Then we cycle through a message Passing Algorithm 28 1 A C D 1 A C D 1 A C D SparkMemory - Accumulator - Used for GlobalState Repeat
#EUeco3 Then we cycle through a message Passing Algorithm 29 1 A C D 1 A C D 1 A C D SparkMemory - Accumulator - Used for GlobalState All Traversers Halt
 Or Program Terminates Result!
#EUeco3 Example OLAP Traversals 30
#EUeco3 Tinkerpop Spark OLAP Pros/Cons Pros • Every message pass requires only a single shuffle • Edges and edge properties accessible without a step • Very Flexible, Many Provider Specific Shortcuts possible • Internal properties can be any Java type • All in one, Server already ready for multiple clients Cons • Limited in ability to connect to external sources/other spark applications • Flexibility of framework allows for many platform specific shortcuts to be added • Genericness provides difficulty in making some optimizations • Edges co-partitioned with vertices, high degree nodes can cause memory issues 31
#EUeco3 GraphFrames Background • Third Party Package • https://graphframes.github.io/ • Integrates with Dataset/Dataframe in Spark • Relational under the hood 32
#EUeco3 GraphFrames are built of two DataFrames 33 Row Column
#EUeco3 GraphFrames are built of two DataFrames 34 id job species Geordi Chief Engineer Human Data Science Officer Android Vertex DataFrame src dst relationship Geordi Data Friend Edge DataFrame Friend
#EUeco3 GraphFrames are built of two DataFrames 35 id job species Geordi Chief Engineer Human Data Science Officer Android Vertex DataFrame src dst relationship Geordi Data Friend Edge DataFrame Friend Can Only Be Spark Types
#EUeco3 GraphFrames are built of two DataFrames 36 id job species Geordi Chief Engineer Human Data Science Officer Android Vertex DataFrame src dst relationship Geordi Data Friend Edge DataFrame Friend No Built in Labels
#EUeco3 Catalyst Optimizes any Requests • Simple requests using DataFrame api don't do anything special • Some methods fall back to GraphX (RDD Based) • Others use pure DataFrame methods 37
#EUeco3 GraphFrames Motif Matching 38 GraphFrame (a)-[e]->(b) V E
#EUeco3 GraphFrames Motif Matching 39 GraphFrame (a)-[e]->(b) Vertex (a) Vertices as a UDT "A"V E A: <VertexRow>
#EUeco3 GraphFrames Motif Matching 40 GraphFrame (a)-[e]->(b) Vertex (a) Vertices as a UDT "A" Edge [b] 
 Edges as UDT "E"
 Join with edges where A.id = E.src V E A: <VertexRow> Join A: <VertexRow>, E: <EdgeRow>
#EUeco3 GraphFrames Motif Matching 41 GraphFrame (a)-[e]->(b) Vertex (a) Vertices as a UDT "A" [e] Vertices as UDT "B" Join with edges where E.dst = B.id Edge Vertex [b] 
 Edges as UDT "E"
 Join with edges where A.id = E.src V E A: <VertexRow> A: <VertexRow>, E: <EdgeRow> Join JoinA: <VertexRow>, E: <EdgeRow>, B: <VertexRow>
#EUeco3 GraphFrames Motif Matching 42 GraphFrame (a)-[e]->(b) Vertex (a) Vertices as a UDT "A" [e] Vertices as UDT "B" Join with edges where E.dst = B.id Edge Vertex [b] 
 Edges as UDT "E"
 Join with edges where A.id = E.src V E A: <VertexRow> A: <VertexRow>, E: <EdgeRow> Join JoinA: <VertexRow>, E: <EdgeRow>, B: <VertexRow> THAT'S SO MANY JOINS
#EUeco3 43 Vertex Edge Vertex A: <VertexRow> A: <VertexRow>, E: <EdgeRow> A: <VertexRow>, E: <EdgeRow>, B: <VertexRow> DataFrames means Optimizations are Automatic
#EUeco3 44 Vertex Edge Vertex A: <VertexRow> A: <VertexRow>, E: <EdgeRow> A: <VertexRow>, E: <EdgeRow>, B: <VertexRow> Select A.ID Columns Pruned and Predicates Pushed
45 Vertex Edge Vertex A: <VertexRow> A: <VertexRow>, E: <EdgeRow> A: <VertexRow>, E: <EdgeRow>, B: <VertexRow> Select A.ID Columns Pruned and Predicates Pushed #EUeco3
46 Vertex Edge Vertex A: <VertexRow> A: <VertexRow>, E: <EdgeRow> A: <VertexRow>, E: <EdgeRow>, B: <VertexRow> Select A.ID Columns Pruned and Predicates Pushed #EUeco3
47 Vertex Edge Vertex A: <VertexRow> A: <VertexRow>, E: <EdgeRow> A: <VertexRow>, E: <EdgeRow>, B: <VertexRow> Select A.ID Columns Pruned and Predicates Pushed #EUeco3
#EUeco3 All of the normal optimizations happen within this FrameWork 48 Vertex Edge Vertex A: <VertexRow> A: <VertexRow>, E: <EdgeRow> A: <VertexRow>, E: <EdgeRow>, B: <VertexRow> Broadcast? Broadcast?
#EUeco3 Code Generation and Internal Rows 49 Vertex Edge Vertex A: <VertexRow> A: <VertexRow>, E: <EdgeRow> A: <VertexRow>, E: <EdgeRow>, B: <VertexRow> Code Generation Code Generation Code Generation Code Generation Code Generation
#EUeco3 GraphFrames Examples 50
#EUeco3 GraphFrame Pros Cons Pros • Much Faster on basic counts • Powerful optimizations + CodeGen • Easy to connect to other sources 
 Cons • Slower on complex traversals (2 Joins per hop) • Relational Model not as Flexible 51
#EUeco3 Choosing the Right Framework 52
Choose TinkerPop OLAP For Long Paths • More complicated queries • Traversals that require many hops • g.V().out.out.out.out 
 • Avoid for simple counts and aggregations • Avoid if you have very high degree Vertices 53#EUeco3
Choose GraphFrames for Interoperability and Short Paths • General Edge/Vertex stats groupCount, min, max • Connecting to other sources • Short paths • High Degree Vertices • Avoid • Long path algorithms 54#EUeco3
#EUeco3 Choosing the Right Framework 55 Gremlin on
 Graphframes OLTP backed by DSE Graph Built in Spark We write it! Search Built In! Advanced Security
#EUeco3 Thanks for Listening 56 Datastax Academy Graph Course https://academy.datastax.com/resources/ds330-datastax-enterprise-graph
 Try out Datastax Enterprise! https://academy.datastax.com/quick-downloads
 
 Apache Tinkerpop
 http://tinkerpop.apache.org/ 
 GraphFrames Link https://graphframes.github.io/

More Related Content

PDF
Cosco: An Efficient Facebook-Scale Shuffle Service
byDatabricks
 
PPT
Parquet overview
byJulien Le Dem
 
PDF
Introduction to Apache Spark
byAnastasios Skarlatidis
 
PDF
The Parquet Format and Performance Optimization Opportunities
byDatabricks
 
PDF
Lessons from Running Large Scale Spark Workloads
byDatabricks
 
PPTX
Introduction to spark
byHome
 
PPTX
Apache Spark overview
byDataArt
 
PPTX
Optimizing Apache Spark SQL Joins
byDatabricks
 
Cosco: An Efficient Facebook-Scale Shuffle Service
byDatabricks
 
Parquet overview
byJulien Le Dem
 
Introduction to Apache Spark
byAnastasios Skarlatidis
 
The Parquet Format and Performance Optimization Opportunities
byDatabricks
 
Lessons from Running Large Scale Spark Workloads
byDatabricks
 
Introduction to spark
byHome
 
Apache Spark overview
byDataArt
 
Optimizing Apache Spark SQL Joins
byDatabricks
 

What's hot

PDF
Spark SQL Deep Dive @ Melbourne Spark Meetup
byDatabricks
 
PDF
Large-Scale Text Processing Pipeline with Spark ML and GraphFrames: Spark Sum...
bySpark Summit
 
PDF
Introduction to spark
byDuyhai Doan
 
PDF
A Deep Dive into Query Execution Engine of Spark SQL
byDatabricks
 
PDF
Apache spark - Architecture , Overview & libraries
byWalaa Hamdy Assy
 
PDF
Improving SparkSQL Performance by 30%: How We Optimize Parquet Pushdown and P...
byDatabricks
 
PDF
Apache Spark Core – Practical Optimization
byDatabricks
 
PDF
Data Engineer's Lunch #83: Strategies for Migration to Apache Iceberg
byAnant Corporation
 
PDF
Parallelizing with Apache Spark in Unexpected Ways
byDatabricks
 
PDF
Big Data Architecture
byGuido Schmutz
 
PDF
Apache Sqoop: A Data Transfer Tool for Hadoop
byCloudera, Inc.
 
PDF
Hadoop Overview & Architecture
byEMC
 
PDF
Spark SQL
byJoud Khattab
 
PDF
Kappa vs Lambda Architectures and Technology Comparison
byKai Wähner
 
PDF
Spark shuffle introduction
bycolorant
 
PPTX
Introduction to Data Engineering
byVivek Aanand Ganesan
 
PPTX
What Is Apache Spark? | Introduction To Apache Spark | Apache Spark Tutorial ...
bySimplilearn
 
PPTX
Introduction to Apache Spark
byRahul Jain
 
PPSX
Hadoop
byNishant Gandhi
 
PPTX
Hadoop Architecture | HDFS Architecture | Hadoop Architecture Tutorial | HDFS...
bySimplilearn
 
Spark SQL Deep Dive @ Melbourne Spark Meetup
byDatabricks
 
Large-Scale Text Processing Pipeline with Spark ML and GraphFrames: Spark Sum...
bySpark Summit
 
Introduction to spark
byDuyhai Doan
 
A Deep Dive into Query Execution Engine of Spark SQL
byDatabricks
 
Apache spark - Architecture , Overview & libraries
byWalaa Hamdy Assy
 
Improving SparkSQL Performance by 30%: How We Optimize Parquet Pushdown and P...
byDatabricks
 
Apache Spark Core – Practical Optimization
byDatabricks
 
Data Engineer's Lunch #83: Strategies for Migration to Apache Iceberg
byAnant Corporation
 
Parallelizing with Apache Spark in Unexpected Ways
byDatabricks
 
Big Data Architecture
byGuido Schmutz
 
Apache Sqoop: A Data Transfer Tool for Hadoop
byCloudera, Inc.
 
Hadoop Overview & Architecture
byEMC
 
Spark SQL
byJoud Khattab
 
Kappa vs Lambda Architectures and Technology Comparison
byKai Wähner
 
Spark shuffle introduction
bycolorant
 
Introduction to Data Engineering
byVivek Aanand Ganesan
 
What Is Apache Spark? | Introduction To Apache Spark | Apache Spark Tutorial ...
bySimplilearn
 
Introduction to Apache Spark
byRahul Jain
 
Hadoop
byNishant Gandhi
 
Hadoop Architecture | HDFS Architecture | Hadoop Architecture Tutorial | HDFS...
bySimplilearn
 

Viewers also liked

PDF
Accelerating Shuffle: A Tailor-Made RDMA Solution for Apache Spark with Yuval...
bySpark Summit
 
PDF
How to Share State Across Multiple Apache Spark Jobs using Apache Ignite with...
bySpark Summit
 
PPTX
Deep-Dive into Deep Learning Pipelines with Sue Ann Hong and Tim Hunter
byDatabricks
 
PDF
FPGA-Based Acceleration Architecture for Spark SQL Qi Xie and Quanfu Wang
bySpark Summit
 
PDF
Storage Engine Considerations for Your Apache Spark Applications with Mladen ...
bySpark Summit
 
PDF
Apache Spark—Apache HBase Connector: Feature Rich and Efficient Access to HBa...
bySpark Summit
 
PDF
Improving Traffic Prediction Using Weather Data with Ramya Raghavendra
bySpark Summit
 
PDF
Indicium: Interactive Querying at Scale Using Apache Spark, Zeppelin, and Spa...
bySpark Summit
 
PDF
Apache Spark and Tensorflow as a Service with Jim Dowling
bySpark Summit
 
PDF
Feature Hashing for Scalable Machine Learning with Nick Pentreath
bySpark Summit
 
PDF
Building Machine Learning Algorithms on Apache Spark with William Benton
bySpark Summit
 
PDF
No More Cumbersomeness: Automatic Predictive Modeling on Apache Spark Marcin ...
bySpark Summit
 
PDF
Histogram Equalized Heat Maps from Log Data via Apache Spark with Arvind Rao
bySpark Summit
 
PDF
Experimental Design for Distributed Machine Learning with Myles Baker
byDatabricks
 
PDF
MMLSpark: Lessons from Building a SparkML-Compatible Machine Learning Library...
bySpark Summit
 
PDF
Art of Feature Engineering for Data Science with Nabeel Sarwar
bySpark Summit
 
PPTX
Low Touch Machine Learning with Leah McGuire (Salesforce)
bySpark Summit
 
PDF
Natural Language Understanding at Scale with Spark-Native NLP, Spark ML, and ...
bySpark Summit
 
Accelerating Shuffle: A Tailor-Made RDMA Solution for Apache Spark with Yuval...
bySpark Summit
 
How to Share State Across Multiple Apache Spark Jobs using Apache Ignite with...
bySpark Summit
 
Deep-Dive into Deep Learning Pipelines with Sue Ann Hong and Tim Hunter
byDatabricks
 
FPGA-Based Acceleration Architecture for Spark SQL Qi Xie and Quanfu Wang
bySpark Summit
 
Storage Engine Considerations for Your Apache Spark Applications with Mladen ...
bySpark Summit
 
Apache Spark—Apache HBase Connector: Feature Rich and Efficient Access to HBa...
bySpark Summit
 
Improving Traffic Prediction Using Weather Data with Ramya Raghavendra
bySpark Summit
 
Indicium: Interactive Querying at Scale Using Apache Spark, Zeppelin, and Spa...
bySpark Summit
 
Apache Spark and Tensorflow as a Service with Jim Dowling
bySpark Summit
 
Feature Hashing for Scalable Machine Learning with Nick Pentreath
bySpark Summit
 
Building Machine Learning Algorithms on Apache Spark with William Benton
bySpark Summit
 
No More Cumbersomeness: Automatic Predictive Modeling on Apache Spark Marcin ...
bySpark Summit
 
Histogram Equalized Heat Maps from Log Data via Apache Spark with Arvind Rao
bySpark Summit
 
Experimental Design for Distributed Machine Learning with Myles Baker
byDatabricks
 
MMLSpark: Lessons from Building a SparkML-Compatible Machine Learning Library...
bySpark Summit
 
Art of Feature Engineering for Data Science with Nabeel Sarwar
bySpark Summit
 
Low Touch Machine Learning with Leah McGuire (Salesforce)
bySpark Summit
 
Natural Language Understanding at Scale with Spark-Native NLP, Spark ML, and ...
bySpark Summit
 

Similar to A Tale of Two Graph Frameworks on Spark: GraphFrames and Tinkerpop OLAP Artem Aliev and Russell Spitzer

PPTX
Web-Scale Graph Analytics with Apache Spark with Tim Hunter
byDatabricks
 
PDF
Web-Scale Graph Analytics with Apache® Spark™
byDatabricks
 
PPTX
Large Scale Graph Analytics with JanusGraph
byP. Taylor Goetz
 
PPTX
Large Scale Graph Analytics with JanusGraph
byDataWorks Summit
 
PDF
Challenging Web-Scale Graph Analytics with Apache Spark
byDatabricks
 
PPTX
Lecture 2.3.1 Graph.pptx
byking779879
 
PDF
Applying large scale text analytics with graph databases
byData Ninja API
 
PDF
Introduction to Graph Databases
byPaolo Pareti
 
PDF
Distributed graph processing
byBartosz Konieczny
 
PDF
Introduction to Property Graph Features (AskTOM Office Hours part 1)
byJean Ihm
 
PDF
The Graph Abstract Data Type-DATA STRUCTURE.pdf
byArchana Gopinath
 
PPTX
ppt 1.pptx
byShasidharaniD
 
PDF
Web-Scale Graph Analytics with Apache® Spark™
byDatabricks
 
PDF
Challenging Web-Scale Graph Analytics with Apache Spark with Xiangrui Meng
byDatabricks
 
PDF
TinkerPop: a story of graphs, DBs, and graph DBs
byJoshua Shinavier
 
PDF
Accumulo Summit 2014: Open Source Graph Analysis and Visualization powered by...
byAccumulo Summit
 
PDF
09_Graphs_handout.pdf
byIsrar63
 
PDF
Data Summer Conf 2018, “Analysing Billion Node Graphs (ENG)” — Giorgi Jvaridz...
byProvectus
 
PDF
A Graph is a Graph is a Graph: Equivalence, Transformation, and Composition o...
byJoshua Shinavier
 
PDF
En un mundo hiperconectado, las bases de datos de grafos son tu arma secreta
byjavier ramirez
 
Web-Scale Graph Analytics with Apache Spark with Tim Hunter
byDatabricks
 
Web-Scale Graph Analytics with Apache® Spark™
byDatabricks
 
Large Scale Graph Analytics with JanusGraph
byP. Taylor Goetz
 
Large Scale Graph Analytics with JanusGraph
byDataWorks Summit
 
Challenging Web-Scale Graph Analytics with Apache Spark
byDatabricks
 
Lecture 2.3.1 Graph.pptx
byking779879
 
Applying large scale text analytics with graph databases
byData Ninja API
 
Introduction to Graph Databases
byPaolo Pareti
 
Distributed graph processing
byBartosz Konieczny
 
Introduction to Property Graph Features (AskTOM Office Hours part 1)
byJean Ihm
 
The Graph Abstract Data Type-DATA STRUCTURE.pdf
byArchana Gopinath
 
ppt 1.pptx
byShasidharaniD
 
Web-Scale Graph Analytics with Apache® Spark™
byDatabricks
 
Challenging Web-Scale Graph Analytics with Apache Spark with Xiangrui Meng
byDatabricks
 
TinkerPop: a story of graphs, DBs, and graph DBs
byJoshua Shinavier
 
Accumulo Summit 2014: Open Source Graph Analysis and Visualization powered by...
byAccumulo Summit
 
09_Graphs_handout.pdf
byIsrar63
 
Data Summer Conf 2018, “Analysing Billion Node Graphs (ENG)” — Giorgi Jvaridz...
byProvectus
 
A Graph is a Graph is a Graph: Equivalence, Transformation, and Composition o...
byJoshua Shinavier
 
En un mundo hiperconectado, las bases de datos de grafos son tu arma secreta
byjavier ramirez
 

More from Spark Summit

PDF
Apache Spark-Bench: Simulate, Test, Compare, Exercise, and Yes, Benchmark wit...
bySpark Summit
 
PDF
Getting Ready to Use Redis with Apache Spark with Dvir Volk
bySpark Summit
 
PDF
Next CERN Accelerator Logging Service with Jakub Wozniak
bySpark Summit
 
PDF
Spline: Apache Spark Lineage not Only for the Banking Industry with Marek Nov...
bySpark Summit
 
PDF
Hardware Acceleration of Apache Spark on Energy-Efficient FPGAs with Christof...
bySpark Summit
 
PDF
Apache Spark and Tensorflow as a Service with Jim Dowling
bySpark Summit
 
PDF
MatFast: In-Memory Distributed Matrix Computation Processing and Optimization...
bySpark Summit
 
PDF
VEGAS: The Missing Matplotlib for Scala/Apache Spark with DB Tsai and Roger M...
bySpark Summit
 
PDF
Improving Traffic Prediction Using Weather Datawith Ramya Raghavendra
bySpark Summit
 
PDF
Deduplication and Author-Disambiguation of Streaming Records via Supervised M...
bySpark Summit
 
PDF
How Nielsen Utilized Databricks for Large-Scale Research and Development with...
bySpark Summit
 
PDF
Goal Based Data Production with Sim Simeonov
bySpark Summit
 
PDF
Apache Spark Structured Streaming Helps Smart Manufacturing with Xiaochang Wu
bySpark Summit
 
PDF
Hiding Apache Spark Complexity for Fast Prototyping of Big Data Applications—...
bySpark Summit
 
PDF
Smack Stack and Beyond—Building Fast Data Pipelines with Jorg Schad
bySpark Summit
 
PDF
Running Spark Inside Containers with Haohai Ma and Khalid Ahmed
bySpark Summit
 
PDF
Best Practices for Using Alluxio with Apache Spark with Gene Pang
bySpark Summit
 
PDF
Powering a Startup with Apache Spark with Kevin Kim
bySpark Summit
 
PDF
Preventing Revenue Leakage and Monitoring Distributed Systems with Machine Le...
bySpark Summit
 
PDF
Variant-Apache Spark for Bioinformatics with Piotr Szul
bySpark Summit
 
Apache Spark-Bench: Simulate, Test, Compare, Exercise, and Yes, Benchmark wit...
bySpark Summit
 
Getting Ready to Use Redis with Apache Spark with Dvir Volk
bySpark Summit
 
Next CERN Accelerator Logging Service with Jakub Wozniak
bySpark Summit
 
Spline: Apache Spark Lineage not Only for the Banking Industry with Marek Nov...
bySpark Summit
 
Hardware Acceleration of Apache Spark on Energy-Efficient FPGAs with Christof...
bySpark Summit
 
Apache Spark and Tensorflow as a Service with Jim Dowling
bySpark Summit
 
MatFast: In-Memory Distributed Matrix Computation Processing and Optimization...
bySpark Summit
 
VEGAS: The Missing Matplotlib for Scala/Apache Spark with DB Tsai and Roger M...
bySpark Summit
 
Improving Traffic Prediction Using Weather Datawith Ramya Raghavendra
bySpark Summit
 
Deduplication and Author-Disambiguation of Streaming Records via Supervised M...
bySpark Summit
 
How Nielsen Utilized Databricks for Large-Scale Research and Development with...
bySpark Summit
 
Goal Based Data Production with Sim Simeonov
bySpark Summit
 
Apache Spark Structured Streaming Helps Smart Manufacturing with Xiaochang Wu
bySpark Summit
 
Hiding Apache Spark Complexity for Fast Prototyping of Big Data Applications—...
bySpark Summit
 
Smack Stack and Beyond—Building Fast Data Pipelines with Jorg Schad
bySpark Summit
 
Running Spark Inside Containers with Haohai Ma and Khalid Ahmed
bySpark Summit
 
Best Practices for Using Alluxio with Apache Spark with Gene Pang
bySpark Summit
 
Powering a Startup with Apache Spark with Kevin Kim
bySpark Summit
 
Preventing Revenue Leakage and Monitoring Distributed Systems with Machine Le...
bySpark Summit
 
Variant-Apache Spark for Bioinformatics with Piotr Szul
bySpark Summit
 

Recently uploaded

PPTX
Presentation 1-PUM Singapore 2015- 2D.pptx
byWilliamChong71
 
PDF
[DSC Europe 25] Nikola Rajovic - Hardware Technologies Under the Hood: RISC-V...
byDataScienceConferenc1
 
PPT
Chapter 3 Reusable Technoldsadadsdsdasdogy.ppt
bytuananle9897
 
PPTX
essential in computer science structure and other
byhncseven7
 
PDF
Introducing Cohesity NetBackup (2).pdf
byNestorNavarro20
 
PPTX
[DSC Europe 25] Boris Perkovic - Lost in performance.pptx
byDataScienceConferenc1
 
PPTX
Presentation 2 - USE OF EC7 IN PLAXIS .pptx
byWilliamChong71
 
PPTX
Copy of First ENGR 100 Presentation.pptx
bygenshinds971
 
PPTX
Betsi- Zero-Trust Security Securing the Modern Enterprise.pptx
bymershaabdisa
 
PPTX
CHAPTER-2hdueidijxjxixidididis%20HTML.pptx
byssuser273bea
 
PPTX
Test Planner for allen 2025-26 samarth batch
byamaanahmed1312
 
PPTX
Safe_Spaces_Act_Modern_Presentation.pptx
byairesmijares2003
 
PPTX
Two Factor Authentication for CSE Students.pptx
byfizarcse
 
PPTX
[DSC Europe 25] Max Talanov - Non digital NNs.pptx
byDataScienceConferenc1
 
PPTX
Customer Segmentation: Seeing the Trees and the Forest Simultaneously - V2
bySK Palu
 
PPTX
[DSC Europe 25] Petar Zivanov - AI meets documents From chatbots to AI-powere...
byDataScienceConferenc1
 
PDF
OPPOTUS - Malaysians on Malaysia (MOM) Q3 2025
byOppotus
 
PPTX
1. Prepare for the application of health survey.pptx
byDegaregeNibret
 
PPTX
[DSC Europe 25] Vid Stimac - Policy Parsimony: Between Oversimplifying and Ov...
byDataScienceConferenc1
 
PPTX
[DSC Europe 25] Bogdan Daniel Maruneac - AI - It starts with you.pptx
byDataScienceConferenc1
 
Presentation 1-PUM Singapore 2015- 2D.pptx
byWilliamChong71
 
[DSC Europe 25] Nikola Rajovic - Hardware Technologies Under the Hood: RISC-V...
byDataScienceConferenc1
 
Chapter 3 Reusable Technoldsadadsdsdasdogy.ppt
bytuananle9897
 
essential in computer science structure and other
byhncseven7
 
Introducing Cohesity NetBackup (2).pdf
byNestorNavarro20
 
[DSC Europe 25] Boris Perkovic - Lost in performance.pptx
byDataScienceConferenc1
 
Presentation 2 - USE OF EC7 IN PLAXIS .pptx
byWilliamChong71
 
Copy of First ENGR 100 Presentation.pptx
bygenshinds971
 
Betsi- Zero-Trust Security Securing the Modern Enterprise.pptx
bymershaabdisa
 
CHAPTER-2hdueidijxjxixidididis%20HTML.pptx
byssuser273bea
 
Test Planner for allen 2025-26 samarth batch
byamaanahmed1312
 
Safe_Spaces_Act_Modern_Presentation.pptx
byairesmijares2003
 
Two Factor Authentication for CSE Students.pptx
byfizarcse
 
[DSC Europe 25] Max Talanov - Non digital NNs.pptx
byDataScienceConferenc1
 
Customer Segmentation: Seeing the Trees and the Forest Simultaneously - V2
bySK Palu
 
[DSC Europe 25] Petar Zivanov - AI meets documents From chatbots to AI-powere...
byDataScienceConferenc1
 
OPPOTUS - Malaysians on Malaysia (MOM) Q3 2025
byOppotus
 
1. Prepare for the application of health survey.pptx
byDegaregeNibret
 
[DSC Europe 25] Vid Stimac - Policy Parsimony: Between Oversimplifying and Ov...
byDataScienceConferenc1
 
[DSC Europe 25] Bogdan Daniel Maruneac - AI - It starts with you.pptx
byDataScienceConferenc1
 

A Tale of Two Graph Frameworks on Spark: GraphFrames and Tinkerpop OLAP Artem Aliev and Russell Spitzer

  • 1.
    Artem Aliev andRussell Spitzer, DataStax A Tale of Two Graph Frameworks on Spark: 
 GraphFrames and Tinkerpop OLAP #EUeco3
  • 2.
    #EUeco3 Pierrot and Harlequin •Artem • Graph Analytics Expert • Earth • Russell • Distributed Systems Enthusiast • Earth 2
  • 3.
    Tinkerpop and GraphFramesprovide Complimentary Approaches for Graph Analytics DataSet Catalyst GraphFrames 3#EUeco3
  • 4.
    Graphs are Verticesand Edges 4 Vertices are things and edges represent their relations to one another #EUeco3
  • 5.
    Graphs are Verticesand Edges 5 Registry: USS Enterprise (NCC-1701-C) Class: Ambassador Service: 2332[11] – 2344 (12 Years) Registry: USS Enterprise (NCC-1701-D) Class: Galaxy Service: 2363–2371 (8 Years) Registry: USS Enterprise (NCC-1701) Class: Constitution class[6] Service: 2245–2285 (40 Years) Registry: USS Enterprise (NCC-1701-A) Class: Enterprise class[8][9] Service: 2286–2293 (7 Years) #EUeco3
  • 6.
    Graphs are Verticesand Edges 6 Registry: USS Enterprise (NCC-1701-C) Class: Ambassador Service: 2332[11] – 2344 (12 Years) Registry: USS Enterprise (NCC-1701-D) Class: Galaxy Service: 2363–2371 (8 Years) Registry: USS Enterprise (NCC-1701) Class: Constitution class[6] Service: 2245–2285 (40 Years) Registry: USS Enterprise (NCC-1701-A) Class: Enterprise class[8][9] Service: 2286–2293 (7 Years) Vertex Properties #EUeco3
  • 7.
    Graphs are Verticesand Edges 7 Registry: USS Enterprise (NCC-1701-C) Class: Ambassador Service: 2332[11] – 2344 (12 Years) Registry: USS Enterprise (NCC-1701-D) Class: Galaxy Service: 2363–2371 (8 Years) Registry: USS Enterprise (NCC-1701) Class: Constitution class[6] Service: 2245–2285 (40 Years) Registry: USS Enterprise (NCC-1701-A) Class: Enterprise class[8][9] Service: 2286–2293 (7 Years) succeeded by succeeded by succeeded by #EUeco3
  • 8.
    Graphs are Verticesand Edges 8 Registry: USS Enterprise (NCC-1701-C) Class: Ambassador Service: 2332[11] – 2344 (12 Years) Registry: USS Enterprise (NCC-1701-D) Class: Galaxy Service: 2363–2371 (8 Years) Registry: USS Enterprise (NCC-1701) Class: Constitution class[6] Service: 2245–2285 (40 Years) Registry: USS Enterprise (NCC-1701-A) Class: Enterprise class[8][9] Service: 2286–2293 (7 Years) Edge Edge Labelsucceeded by succeeded by succeeded by #EUeco3
  • 9.
    Graphs are Verticesand Edges 9 Registry: USS Enterprise (NCC-1701-C) Class: Ambassador Service: 2332[11] – 2344 (12 Years) Registry: USS Enterprise (NCC-1701-D) Class: Galaxy Service: 2363–2371 (8 Years) Registry: USS Enterprise (NCC-1701) Class: Constitution class[6] Service: 2245–2285 (40 Years) Registry: USS Enterprise (NCC-1701-A) Class: Enterprise class[8][9] Service: 2286–2293 (7 Years) Ship Ship Ship Ship Vertex Label succeeded by succeeded by succeeded by #EUeco3
  • 10.
    Graphs are Verticesand Edges 10 Registry: USS Enterprise (NCC-1701-C) Class: Ambassador Service: 2332[11] – 2344 (12 Years) Registry: USS Enterprise (NCC-1701-D) Class: Galaxy Service: 2363–2371 (8 Years) Registry: USS Enterprise (NCC-1701) Class: Constitution class Service: 2245–2285 (40 Years) Ship Ship Ship Ship Position: Captain
 Name: Kirk Position: Captain
 Name: Picard Crew Crew succeeded by succeeded by succeeded by #EUeco3
  • 11.
    Graphs are Verticesand Edges 11 Registry: USS Enterprise (NCC-1701-C) Class: Ambassador Service: 2332[11] – 2344 (12 Years) Registry: USS Enterprise (NCC-1701-D) Class: Galaxy Service: 2363–2371 (8 Years) Registry: USS Enterprise (NCC-1701) Class: Constitution class Service: 2245–2285 (40 Years) Registry: USS Enterprise (NCC-1701-A) Class: Enterprise class Service: 2286–2293 (7 Years) Ship Ship Ship Ship Position: Captain
 Name: Kirk Position: Captain
 Name: Picard Crew Crew succeeded by succeeded by succeeded by served on served on served on served on #EUeco3
  • 12.
    Graphs are Verticesand Edges 12 Registry: USS Enterprise (NCC-1701-C) Class: Ambassador Service: 2332[11] – 2344 (12 Years) Registry: USS Enterprise (NCC-1701-D) Class: Galaxy Service: 2363–2371 (8 Years) Registry: USS Enterprise (NCC-1701) Class: Constitution class Service: 2245–2285 (40 Years) Registry: USS Enterprise (NCC-1701-A) Class: Enterprise class Service: 2286–2293 (7 Years) Ship Ship Ship Ship Position: Captain
 Name: Kirk Position: Captain
 Name: Picard Crew Crew succeeded by succeeded by succeeded by served on served on served on served on But why do I want this? #EUeco3
  • 13.
    Graphs let usask questions about our data based on their relations 13 What Captain Served After Kirk? What Ship was two after the NCC-1701? #EUeco3
  • 14.
    Traversals involve followingpaths through the Graph 14 Registry: USS Enterprise (NCC-1701-C) Class: Ambassador Service: 2332[11] – 2344 (12 Years) Registry: USS Enterprise (NCC-1701-D) Class: Galaxy Service: 2363–2371 (8 Years) Registry: USS Enterprise (NCC-1701) Class: Constitution class Service: 2245–2285 (40 Years) Registry: USS Enterprise (NCC-1701-A) Class: Enterprise class Service: 2286–2293 (7 Years) Ship Ship Ship Ship Position: Captain
 Name: Kirk Position: Captain
 Name: Picard Crew Crew succeeded by succeeded by succeeded by served on served on served on served on #EUeco3
  • 15.
    What Captain wasAfter Kirk? 15 Registry: USS Enterprise (NCC-1701-C) Class: Ambassador Service: 2332[11] – 2344 (12 Years) Registry: USS Enterprise (NCC-1701-A) Class: Enterprise class Service: 2286–2293 (7 Years) Ship Ship Position: Captain
 Name: Kirk Position: Captain
 Name: Picard Crew Crewsucceeded by served on served on #EUeco3
  • 16.
    What Ship wastwo after the NCC-1701? 16 Registry: USS Enterprise (NCC-1701-C) Class: Ambassador Service: 2332[11] – 2344 (12 Years) Registry: USS Enterprise (NCC-1701) Class: Constitution class Service: 2245–2285 (40 Years) Registry: USS Enterprise (NCC-1701-A) Class: Enterprise class Service: 2286–2293 (7 Years) Ship Ship Ship succeeded by succeeded by #EUeco3
  • 17.
    Tinkerpop is aPowerful and Flexible Graph Framework • Server, Language, Connectors • Graph Framework for 
 OLAP and OLTP • Node Centric Representations • Fluent API (Gremlin) • Fully Self Contained Framework 17#EUeco3
  • 18.
  • 19.
  • 20.
  • 21.
    #EUeco3 What happens whenyou have too much data? 21
  • 22.
    #EUeco3 Tinkerpop Spark OLAPMechanism • Instead of one traversal we traverse starting from all nodes simultaneously 22
  • 23.
    Distribution Requires Partitioning 23 ? BigData Independent Chunks of Data#EUeco3
  • 24.
    #EUeco3 Vertex Stored ina PairRDD Id -> StarVertex(Edge and Property Information) 24 1 A C D Star Vertex: Adjacency list representation
 1: "A", "Kirk"
 A: "C", "Kirk"
 C: "D", "Picard"
 D: "Picard"
 Just Id 
 Of Connected 
 Vertex
  • 25.
    #EUeco3 Vertex Program RunsInitializing Traverser for every Vertex 25 1 A C D SparkMemory - Accumulator - Used for GlobalState
  • 26.
    #EUeco3 Then we cyclethrough a message Passing Algorithm 26 1 A C D 1 A C D 1 A C D SparkMemory - Accumulator - Used for GlobalState
  • 27.
    #EUeco3 Then we cyclethrough a message Passing Algorithm 27 1 A C D 1 A C D 1 A C D SparkMemory - Accumulator - Used for GlobalState Passes messages from one Vertex to another with a join
  • 28.
    #EUeco3 Then we cyclethrough a message Passing Algorithm 28 1 A C D 1 A C D 1 A C D SparkMemory - Accumulator - Used for GlobalState Repeat
  • 29.
    #EUeco3 Then we cyclethrough a message Passing Algorithm 29 1 A C D 1 A C D 1 A C D SparkMemory - Accumulator - Used for GlobalState All Traversers Halt
 Or Program Terminates Result!
  • 30.
  • 31.
    #EUeco3 Tinkerpop Spark OLAPPros/Cons Pros • Every message pass requires only a single shuffle • Edges and edge properties accessible without a step • Very Flexible, Many Provider Specific Shortcuts possible • Internal properties can be any Java type • All in one, Server already ready for multiple clients Cons • Limited in ability to connect to external sources/other spark applications • Flexibility of framework allows for many platform specific shortcuts to be added • Genericness provides difficulty in making some optimizations • Edges co-partitioned with vertices, high degree nodes can cause memory issues 31
  • 32.
    #EUeco3 GraphFrames Background • ThirdParty Package • https://graphframes.github.io/ • Integrates with Dataset/Dataframe in Spark • Relational under the hood 32
  • 33.
    #EUeco3 GraphFrames are builtof two DataFrames 33 Row Column
  • 34.
    #EUeco3 GraphFrames are builtof two DataFrames 34 id job species Geordi Chief Engineer Human Data Science Officer Android Vertex DataFrame src dst relationship Geordi Data Friend Edge DataFrame Friend
  • 35.
    #EUeco3 GraphFrames are builtof two DataFrames 35 id job species Geordi Chief Engineer Human Data Science Officer Android Vertex DataFrame src dst relationship Geordi Data Friend Edge DataFrame Friend Can Only Be Spark Types
  • 36.
    #EUeco3 GraphFrames are builtof two DataFrames 36 id job species Geordi Chief Engineer Human Data Science Officer Android Vertex DataFrame src dst relationship Geordi Data Friend Edge DataFrame Friend No Built in Labels
  • 37.
    #EUeco3 Catalyst Optimizes anyRequests • Simple requests using DataFrame api don't do anything special • Some methods fall back to GraphX (RDD Based) • Others use pure DataFrame methods 37
  • 38.
  • 39.
  • 40.
    #EUeco3 GraphFrames Motif Matching 40 GraphFrame (a)-[e]->(b) Vertex(a) Vertices as a UDT "A" Edge [b] 
 Edges as UDT "E"
 Join with edges where A.id = E.src V E A: <VertexRow> Join A: <VertexRow>, E: <EdgeRow>
  • 41.
    #EUeco3 GraphFrames Motif Matching 41 GraphFrame (a)-[e]->(b) Vertex(a) Vertices as a UDT "A" [e] Vertices as UDT "B" Join with edges where E.dst = B.id Edge Vertex [b] 
 Edges as UDT "E"
 Join with edges where A.id = E.src V E A: <VertexRow> A: <VertexRow>, E: <EdgeRow> Join JoinA: <VertexRow>, E: <EdgeRow>, B: <VertexRow>
  • 42.
    #EUeco3 GraphFrames Motif Matching 42 GraphFrame (a)-[e]->(b) Vertex(a) Vertices as a UDT "A" [e] Vertices as UDT "B" Join with edges where E.dst = B.id Edge Vertex [b] 
 Edges as UDT "E"
 Join with edges where A.id = E.src V E A: <VertexRow> A: <VertexRow>, E: <EdgeRow> Join JoinA: <VertexRow>, E: <EdgeRow>, B: <VertexRow> THAT'S SO MANY JOINS
  • 43.
    #EUeco3 43 Vertex Edge Vertex A: <VertexRow> A:<VertexRow>, E: <EdgeRow> A: <VertexRow>, E: <EdgeRow>, B: <VertexRow> DataFrames means Optimizations are Automatic
  • 44.
    #EUeco3 44 Vertex Edge Vertex A: <VertexRow> A:<VertexRow>, E: <EdgeRow> A: <VertexRow>, E: <EdgeRow>, B: <VertexRow> Select A.ID Columns Pruned and Predicates Pushed
  • 45.
    45 Vertex Edge Vertex A: <VertexRow> A: <VertexRow>, E:<EdgeRow> A: <VertexRow>, E: <EdgeRow>, B: <VertexRow> Select A.ID Columns Pruned and Predicates Pushed #EUeco3
  • 46.
    46 Vertex Edge Vertex A: <VertexRow> A: <VertexRow>, E:<EdgeRow> A: <VertexRow>, E: <EdgeRow>, B: <VertexRow> Select A.ID Columns Pruned and Predicates Pushed #EUeco3
  • 47.
    47 Vertex Edge Vertex A: <VertexRow> A: <VertexRow>, E:<EdgeRow> A: <VertexRow>, E: <EdgeRow>, B: <VertexRow> Select A.ID Columns Pruned and Predicates Pushed #EUeco3
  • 48.
    #EUeco3 All of thenormal optimizations happen within this FrameWork 48 Vertex Edge Vertex A: <VertexRow> A: <VertexRow>, E: <EdgeRow> A: <VertexRow>, E: <EdgeRow>, B: <VertexRow> Broadcast? Broadcast?
  • 49.
    #EUeco3 Code Generation andInternal Rows 49 Vertex Edge Vertex A: <VertexRow> A: <VertexRow>, E: <EdgeRow> A: <VertexRow>, E: <EdgeRow>, B: <VertexRow> Code Generation Code Generation Code Generation Code Generation Code Generation
  • 50.
  • 51.
    #EUeco3 GraphFrame Pros Cons Pros •Much Faster on basic counts • Powerful optimizations + CodeGen • Easy to connect to other sources 
 Cons • Slower on complex traversals (2 Joins per hop) • Relational Model not as Flexible 51
  • 52.
  • 53.
    Choose TinkerPop OLAPFor Long Paths • More complicated queries • Traversals that require many hops • g.V().out.out.out.out 
 • Avoid for simple counts and aggregations • Avoid if you have very high degree Vertices 53#EUeco3
  • 54.
    Choose GraphFrames forInteroperability and Short Paths • General Edge/Vertex stats groupCount, min, max • Connecting to other sources • Short paths • High Degree Vertices • Avoid • Long path algorithms 54#EUeco3
  • 55.
    #EUeco3 Choosing the RightFramework 55 Gremlin on
 Graphframes OLTP backed by DSE Graph Built in Spark We write it! Search Built In! Advanced Security
  • 56.
    #EUeco3 Thanks for Listening 56 DatastaxAcademy Graph Course https://academy.datastax.com/resources/ds330-datastax-enterprise-graph
 Try out Datastax Enterprise! https://academy.datastax.com/quick-downloads
 
 Apache Tinkerpop
 http://tinkerpop.apache.org/ 
 GraphFrames Link https://graphframes.github.io/