Apache Spark is a powerful framework for distributed computing. You can you its API to create connectors for a variety of data formats, both in reading and writing. In this session I'll introduce you to the official Neo4j Connector for Apache Spark that allows a bi-directional communication between these two tools. We'll discuss the challenges that we faced on our trip to the first release of the connector, and we'll see how to leverage Neo4j alongside the computational power of Spark.

1. Davide Fantuzzi - Data Engineer
The Spark of Neo4j
25 Febbraio 2021 - GraphRM

2. WELCOME
Davide Fantuzzi
Data Engineer @ LARUS Business Automation
PICTURE
@utnaf
/in/davidefantuzzi/

3. ABOUT LARUS
Founded in 2004
HQ: Venice
Offices: Pescara, Rome, Milan
Global services
International projects
Data Engineer, Data
Architect, Data Scientist, Big
Data certified experts team
We help companies to
become insight-driven
organizations
Leader in development of
data-driven application
based on NoSQL & Event
Streaming Technologies.

4. LARUS: OUR SPECIALTIES
Big Data Platform Design &
Development (Java, Scala,
Python, Javascript)
Data Engineering
Graph Data Visualization
Data Science
Strategic Advisoring for
Data-Driven Transformation
Projects
Machine Learning and AI
graph based technology

5. LARUS NEO4J

6. LARUS: OUR PARTNERS

7. 7
AGENDA
1. Spark & Neo4j
2. Challenges
3. Neo4j Connector for Apache Spark
4. Demo

8. 8
SPARK & NEO4J
GraphRM

9. WHAT IS APACHE SPARK?
GraphRM 25 Febbraio 2021
● Analytics engine for large-scale data
processing
● Cluster of worker nodes which partition
operations and execute in parallel
● Supports SQL & Streaming
● Oriented around DataFrames which for our
purposes are effectively tables
● Is Polyglot

10. WHEN TO USE SPARK?
GraphRM 25 Febbraio 2021
● Very large datasets that have to be broken into pieces
● Complex pipelines with many sources & transformations
● Great for iterative algorithms (map, reduce, filter, sort)

11. OUR GOAL
GraphRM 25 Febbraio 2021
● Avoid custom "hacky" solutions
● Continuous development
● Quick response to issues
● Leverage DataSource V2 APIs in order to be fully Spark Compliant
○ Polyglot
○ ETL
○ Graph-Driven Machine Learning

12. OUR SOLUTION
GraphRM 25 Febbraio 2021
● Deprecation of the old connector
● Complete rewrite using DataSource API V2
● Official (enterprise) Neo4j support through Larus
● Dedicated Team
● Open-source
● Comprehensive Documentation

13. 13
CHALLENGES
GraphRM

14. DOCUMENTATION AND EXAMPLES
GraphRM 25 Febbraio 2021
● Lack of.
● No official documentation on the DataSource V2 API
● Examples on the web were superficial

15. BREAKING CHANGES
GraphRM 25 Febbraio 2021
Spark 2.3 Spark 2.4 Spark 3.0
Breaking Changes Breaking Changes

16. WHICH VERSION TO START WITH?
GraphRM 25 Febbraio 2021
Spark 2.3 Spark 2.4 Spark 3.0
Spark 2.4

18. WHICH VERSION TO START WITH?
GraphRM 25 Febbraio 2021

19. WHICH VERSION TO START WITH?
GraphRM 25 Febbraio 2021
● Spark 2.4 is Supported
● Spark 3.0 is pre-released and an official released will be happening in March
● Spark 2.3 in on hold

20. VERSION FRAGMENTATION
GraphRM 25 Febbraio 2021
● Spark 2.4 supports Scala 2.11 and Scala 2.12
● Spark 3.0 supports Scala 2.12 and Scala 2.13, and removed the support for Scala 2.11
● (but Spark 3.0 for Scala 2.13 is not released yet)
● We need to release a JAR for each combination
neo4j-connector-apache-spark_2.12_3.0-4.0.0.jar

21. VERSION FRAGMENTATION
GraphRM 25 Febbraio 2021
Maven modules to the rescue!

22. TABLES VS LABELS
GraphRM 25 Febbraio 2021
Person Movie
:ACTED_IN
We had to find a way to map Graph entities into
table columns
Person.id Person.name
1 Keanu Reeves
2 Tom Hanks
ACTED_IN.source ACTED_IN.target
1 3
2 4
Movie.id Movie.title
3 The Matrix
4 Cloud Atlas

23. TABLES VS LABELS
GraphRM 25 Febbraio 2021
Bi-directional mapping allows to read from and
write to Neo4j
Person.id Person.name
1 Keanu Reeves
2 Tom Hanks
ACTED_IN.source ACTED_IN.target
1 3
2 4
Movie.id Movie.title
3 The Matrix
4 Cloud Atlas
READ WRITE

24. SCHEMA VS. SCHEMALESS
GraphRM 25 Febbraio 2021
● Result flattening
name String
age Long
location String
name age location
“John Doe” 33 “Milan”
“Jane Doe” 24 “Rome”
Result
Schema

25. SCHEMA VS. SCHEMALESS
GraphRM 25 Febbraio 2021
● Result flattening
● Property might not exist in every
record
name String
age Long
location String
name age location
“John Doe” 33 “Milan”
“Jane Doe” 24 null
Result
Schema

26. SCHEMA VS. SCHEMALESS
Graph RM 25 Febbraio 2021
● Result flattening
● Property might not exist in every
record
● Property might not have type
consistency
name String
age String*
location String
name age location
“John Doe” “33” “Milan”
“Jane Doe” “24” “Rome”
Result
Schema
* no matter the types involved

27. 27
NEO4J CONNECTOR FOR
APACHE SPARK
GraphRM

28. READ DATA
GraphRM 25 Febbraio 2021
● Labels
val df = spark.read.format("org.neo4j.spark.DataSource")
.option("labels", ":Person:Admin")
.load()
df = spark.read.format("org.neo4j.spark.DataSource")
.option("labels", ":Person:Admin")
.load()
df <- read.df(source="org.neo4j.spark.DataSource",
labels=":Person:Admin")
Scala
Python
R

29. READ DATA
GraphRM 25 Febbraio 2021
● Labels
● Relationship
val df = spark.read.format("org.neo4j.spark.DataSource")
.option("relationship", "ACTED_IN")
.option("relationship.source.labels", "Person")
.option("relationship.target.labels", "Movie")
.load()

30. READ DATA
GraphRM 25 Febbraio 2021
● Labels
● Relationship
● Query
val df = spark.read.format("org.neo4j.spark.DataSource")
.option("query","MATCH (n:Person) RETURN n.name, n.age")
.load()

31. WRITE DATA
GraphRM 25 Febbraio 2021
● Labels val bandDf = Seq(
(1, "Alex Lifeson"),
(2, "Neil Peart"),
(3, "Geddy Lee")
).toDF("id", "name")
bandDf.write
.format("org.neo4j.spark.DataSource")
.option("labels", ":Person:Musician")
.save

32. WRITE DATA
GraphRM 25 Febbraio 2021
● Labels
● Relationship
val musicDf = Seq(
(12, "John Bonham", "Drums"),
(19, "John Mayer", "Guitar"),
(32, "John Scofield", "Guitar"),
(15, "John Butler", "Guitar")
).toDF("experience", "name", "instrument")
musicDf.write
.format("org.neo4j.spark.DataSource")
.option("relationship", "PLAYS")
.option("relationship.save.strategy", "keys")
.option("relationship.source.labels", ":Musician")
.option("relationship.source.node.keys", "name:name")
.option("relationship.target.labels", ":Instrument")
.option("relationship.target.node.keys", "instrument:name")
.save

33. WRITE DATA
GraphRM 25 Febbraio 2021
● Labels
● Relationship
● Query
val theTeam = Seq(
("David", "Allen"),
("Andrea", "Santurbano"),
("Davide", "Fantuzzi")
).toDF("name", "lastname")
theTeam.write
.format("org.neo4j.spark.DataSource")
.option(
"query",
"CREATE (n:Person)" +
"SET fullName = event.name + event.lastname"
)
.save()
This will generate a query like:
UNWIND $events AS event
CREATE (n:Person) SET fullName = event.name + event.lastname

34. FEATURES
GraphRM 25 Febbraio 2021
● Push Down Filters val df = spark.read
.format("org.neo4j.spark.DataSource")
.option("labels", "Movie")
.load
df.where("title LIKE 'Matrix%'").show()

35. FEATURES
GraphRM 25 Febbraio 2021
● Push Down Filters
● Push Down Columns
val df = spark.read
.format("org.neo4j.spark.DataSource")
.option("labels", "Movie")
.load
df.select("title").show()

36. FEATURES
GraphRM 25 Febbraio 2021
● Push Down Filters
● Push Down Columns
● Official Neo4j Driver [link]

37. FEATURES
GraphRM 25 Febbraio 2021
● Push Down Filters
● Push Down Columns
● Official Neo4j Driver [link]
● CypherDSL [link]

38. FEATURES
GraphRM 25 Febbraio 2021
● Push Down Filters
● Push Down Columns
● Official Neo4j Driver [link]
● CypherDSL [link]
● GraphX / GraphFrames are not used

39. COMMON USE CASES
GraphRM 25 Febbraio 2021
● Data Source Integration
○ Connect any supported file format or database of Spark
○ To Neo4j
● Extraction, Transformation, and Load (ETL) bi-directionally
○ Bulk insert for new databases
○ Ongoing nightly jobs
● Graph-driven Machine Learning
○ Use Spark to leverage Graph Data Science to existing pipelines

40. 40
DEMO
GraphRM

41. USEFUL LINKS
GraphRM 25 Febbraio 2021
● GitHub https://github.com/neo4j-contrib/neo4j-spark-connector
● Documentation https://neo4j.com/developer/spark/
● Notebook for playing around with the connector
https://github.com/utnaf/neo4j-connector-apache-spark-notebooks
● Article on Towards Data Science
https://towardsdatascience.com/using-neo4j-with-pyspark-on-databricks-eb3d127f2245

42. THANKS FOR YOUR ATTENTION
Davide Fantuzzi
Data Engineer
@utnaf