Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Big Graph Analytics on Neo4j with
Apache Spark
Kenny Bastani
FOSDEM '15, Graph Processing Devroom
My background
Second year speaking in the graph devroom
Extremely thankful for all the organizers of this
devroom have don...
I apologize in advance for these
incredibly low-budget slides
Engineer + Evangelism
I evangelize about Neo4j and graph database
technology
I am an engineer at Digital Insight, a Silico...
Engineering
NCR/Digital Insight
$18 Trillion per year in ATM withdrawals
As a Graph Database Evangelist
I make cool things with graphs and blog about it
Just so we're clear…
I'm not selling you anything today
Agenda
Let's try and have some fun.
We're going to run PageRank on all human knowledge.
After a lot of low-budget slides.
The Problem
It's hard to analyze graphs at scale
The importance of graph algorithms
PageRank gave us Google
Friend of a friend gave us Facebook
Every hacker and tinkerer should be
able to turn world changing ideas into a
reality
Every research scientist who needs
graph analytics to save millions of lives
should have that power
The simple fact is that you are brilliant
but your brilliant ideas require complex
big data analytics
So then you need to learn a lot of
things or steal a lot of money to change
the world
Why is it so hard to do this stuff?
Enemy #1:
Relational Databases
Relational databases store data in ways that make it
difficult to extract graphs for analys...
– This guy represents every business person that controls your future as an
engineer
"I need to combine 32 different table...
You're probably thinking…
Sorry business, no PageRank for you!
But you courageously move forward
regardless
This is what is likely to happen next.
Enemy #2:
Big Data
If you still think Big Data is a buzz word
You haven't had to feel the pain of failing at it.
When you hit a wall because your data is
too big
You start to see what this big data thing is all about.
What seems to be the problem? Where's
my PageRank
All those records you merged from those 32 different
tables turns out to...
Frantic and scared
You turn to your most dependable friend
You might search for a
Now that you know that Big Data is the
real deal
You read "Big Data for Dummies" and continue to
tackle the PageRank probl...
Distributed File Systems
Distributed file systems are a foundational component
of big data analytics
Chops things into man...
Hadoop MapReduce
Worth mentioning, Hadoop started this whole
MapReduce craze
You could translate the raw data from a CSV a...
Ok so now you know about Big Data,
Hadoop, HDFS
You fire up your Amazon EC2 Hadoop cluster...
This guy is still waiting…
You hold your breath..
And submit the PageRank job…
You wait…
3 hours later…
Out of memory: heap space exceeded!
It must be the configs
You check the configs
Increase the heap space
Do some Stackoverflow trolling
And you submit the Pag...
Graph algorithms can be evil at scale
It depends on the complexity of your graph
How many strongly connected components yo...
It doesn't matter how many nodes you
have in your cluster
For iterative graph algorithms the complexity of the
graph will ...
This guy is going to have to settle for
collaborative filtering
Neo4j Mazerunner Project
What is Neo4j Mazerunner?
The basic idea is…
Graph databases need ETL so you can analyze your
data and look it up later.
Graph databases are great a...
Docker
If you're not up on Docker, let me give you a quick
intro.
Docker
Docker is a VM framework that lets you easily create a
recipe for an image and deploy applications with ease.
The i...
Why?
If I am an engineer on a product team, I want to
choose my own software libraries and languages to
solve problems.
Microservices for the win
So here is the future of software development:
• Cloud OS like Apache Mesos manages datacenter
r...
Microservices cont.
Docker gives you the freedom to use Neo4j, or
OrientDB, or MongoDB or whatever application
dependency ...
Beware of the monolith…
Monolithic apps are those software platforms that just
try and do every possible damn thing. They'...
Docker cont.
Summarizing:
• Docker containerizes your bad engineering
decisions without bringing down your platform.
• So ...
So is this guy…
Mazerunner runs on Docker
You can pull it down and deploy it safely and roll the
dice on some awesome analytics capabiliti...
HDFS and Apache Spark
Apache Spark is a really interesting open source
project.
It is a scalable big data and machine lear...
So I wrote the world's first analysis
service for a graph database that does
2-way ETL
That scared a lot of people in the
"graph databases are infallible" club
Analysis is not a lookup
Analytics on graphs takes massive amounts of
system resources and might bring down your OLTP
capabilities as it competes t...
Now let's fire up Neo4j Mazerunner
Demo Goals:
• I will hopefully be successful at showing you how
to install Mazerunner o...
Where do we go now?
Become a committer to the project and let's make it better
Find the link on my blog — www.kennybastani...
Thanks!
Follow me on Twitter:
http://www.twitter.com/kennybastani
Upcoming SlideShare
Loading in …5
×

Big Graph Analytics on Neo4j with Apache Spark

In this talk I will introduce you to a Docker container that provides you an easy way to do distributed graph processing using Apache Spark GraphX and a Neo4j graph database. You'll learn how to analyze big data graphs that are exported from Neo4j and consequently updated from the results of a Spark GraphX analysis. The types of analysis I will be talking about are PageRank, connected components, triangle counting, and community detection.

Database technologies have evolved to be able to store big data, but are largely inflexible. For complex graph data models stored in a relational database there may be tedious transformations and shuffling around of data to perform large scale analysis.

Fast and scalable analysis of big data has become a critical competitive advantage for companies. There are open source tools like Apache Hadoop and Apache Spark that are providing opportunities for companies to solve these big data problems in a scalable way. Platforms like these have become the foundation of the big data analysis movement.
Speakers

Big Graph Analytics on Neo4j with Apache Spark

  1. 1. Big Graph Analytics on Neo4j with Apache Spark Kenny Bastani FOSDEM '15, Graph Processing Devroom
  2. 2. My background Second year speaking in the graph devroom Extremely thankful for all the organizers of this devroom have done for this technology Thank you organizers!
  3. 3. I apologize in advance for these incredibly low-budget slides
  4. 4. Engineer + Evangelism I evangelize about Neo4j and graph database technology I am an engineer at Digital Insight, a Silicon Valley based SaaS banking platform provider.
  5. 5. Engineering NCR/Digital Insight $18 Trillion per year in ATM withdrawals
  6. 6. As a Graph Database Evangelist I make cool things with graphs and blog about it
  7. 7. Just so we're clear… I'm not selling you anything today
  8. 8. Agenda Let's try and have some fun. We're going to run PageRank on all human knowledge. After a lot of low-budget slides.
  9. 9. The Problem It's hard to analyze graphs at scale
  10. 10. The importance of graph algorithms PageRank gave us Google Friend of a friend gave us Facebook
  11. 11. Every hacker and tinkerer should be able to turn world changing ideas into a reality
  12. 12. Every research scientist who needs graph analytics to save millions of lives should have that power
  13. 13. The simple fact is that you are brilliant but your brilliant ideas require complex big data analytics
  14. 14. So then you need to learn a lot of things or steal a lot of money to change the world
  15. 15. Why is it so hard to do this stuff?
  16. 16. Enemy #1: Relational Databases Relational databases store data in ways that make it difficult to extract graphs for analysis
  17. 17. – This guy represents every business person that controls your future as an engineer "I need to combine 32 different tables in 5 different system of records and put it in a CSV every hour."
  18. 18. You're probably thinking…
  19. 19. Sorry business, no PageRank for you!
  20. 20. But you courageously move forward regardless This is what is likely to happen next.
  21. 21. Enemy #2: Big Data If you still think Big Data is a buzz word You haven't had to feel the pain of failing at it.
  22. 22. When you hit a wall because your data is too big You start to see what this big data thing is all about.
  23. 23. What seems to be the problem? Where's my PageRank All those records you merged from those 32 different tables turns out to be petabytes of data.
  24. 24. Frantic and scared You turn to your most dependable friend
  25. 25. You might search for a
  26. 26. Now that you know that Big Data is the real deal You read "Big Data for Dummies" and continue to tackle the PageRank problem
  27. 27. Distributed File Systems Distributed file systems are a foundational component of big data analytics Chops things into manageable sized blocks, usually 64mb Spreads those blocks out across a cluster of VM resources
  28. 28. Hadoop MapReduce Worth mentioning, Hadoop started this whole MapReduce craze You could translate the raw data from a CSV and turn it into a map of keys to values Keys are distributed per node and used to reduce the values into a partitioned analysis
  29. 29. Ok so now you know about Big Data, Hadoop, HDFS You fire up your Amazon EC2 Hadoop cluster...
  30. 30. This guy is still waiting…
  31. 31. You hold your breath.. And submit the PageRank job… You wait…
  32. 32. 3 hours later… Out of memory: heap space exceeded!
  33. 33. It must be the configs You check the configs Increase the heap space Do some Stackoverflow trolling And you submit the PageRank job again…
  34. 34. Graph algorithms can be evil at scale It depends on the complexity of your graph How many strongly connected components you have But since some graph algorithms like PageRank are iterative You have to iterate from one stage and use the results of the previous stage
  35. 35. It doesn't matter how many nodes you have in your cluster For iterative graph algorithms the complexity of the graph will make you or break you Graphs with high complexity need a lot of memory to be processed iteratively
  36. 36. This guy is going to have to settle for collaborative filtering
  37. 37. Neo4j Mazerunner Project
  38. 38. What is Neo4j Mazerunner?
  39. 39. The basic idea is… Graph databases need ETL so you can analyze your data and look it up later. Graph databases are great and all, but… No platform in the open source world should be the one platform that does everything. Especially a database.
  40. 40. Docker If you're not up on Docker, let me give you a quick intro.
  41. 41. Docker Docker is a VM framework that lets you easily create a recipe for an image and deploy applications with ease. The idea is that infrastructure and operational complexity makes it hard for agile development of new products.
  42. 42. Why? If I am an engineer on a product team, I want to choose my own software libraries and languages to solve problems.
  43. 43. Microservices for the win So here is the future of software development: • Cloud OS like Apache Mesos manages datacenter resources • If you build a new service, use whatever application framework you want. As long as you communicate over REST.
  44. 44. Microservices cont. Docker gives you the freedom to use Neo4j, or OrientDB, or MongoDB or whatever application dependency you want inside your container. Because of something called graceful degradation, if OrientDB or Neo4j fail at being everything, they'll fault only within their container and not bring your entire SaaS platform to its knees.
  45. 45. Beware of the monolith… Monolithic apps are those software platforms that just try and do every possible damn thing. They're like Swiss army knives of the software world. If you rely on one service to do everything, your entire platform is going to come down when it fails. And it will fail…
  46. 46. Docker cont. Summarizing: • Docker containerizes your bad engineering decisions without bringing down your platform. • So I'm pretty much a fan of that.
  47. 47. So is this guy…
  48. 48. Mazerunner runs on Docker You can pull it down and deploy it safely and roll the dice on some awesome analytics capabilities that lend well to graph data models.
  49. 49. HDFS and Apache Spark Apache Spark is a really interesting open source project. It is a scalable big data and machine learning platform. It also lets you do in-memory analytics on a graph dataset.
  50. 50. So I wrote the world's first analysis service for a graph database that does 2-way ETL
  51. 51. That scared a lot of people in the "graph databases are infallible" club
  52. 52. Analysis is not a lookup
  53. 53. Analytics on graphs takes massive amounts of system resources and might bring down your OLTP capabilities as it competes to share system resources
  54. 54. Now let's fire up Neo4j Mazerunner Demo Goals: • I will hopefully be successful at showing you how to install Mazerunner on Docker • I will demo you an analysis job scheduler that extracts subgraphs, analyzes them, and pops the results back to Neo4j
  55. 55. Where do we go now? Become a committer to the project and let's make it better Find the link on my blog — www.kennybastani.com
  56. 56. Thanks! Follow me on Twitter: http://www.twitter.com/kennybastani

×