This document summarizes Michael Hunger's presentation on how graphs make databases fun again. Some key points: - Traditional relational databases have issues modeling connected data and performing complex queries over relationships. Graph databases like Neo4j can more naturally represent connected data as nodes and relationships. - Neo4j was originally created to solve issues modeling connected data for a digital asset management system. It uses a graph data model and allows complex relationship queries through its Cypher query language. - The document demonstrates importing meetup data into Neo4j and running queries to find connections between users, groups, and topics. It also shows examples of querying actor relationships and movie data. - Tools are presented

1. How Graphs make Databases Fun again
Value in Relationships
Michael Hunger (@mesirii)
vJUG July 2015

2. (Michael)-[:HELPS]->(People)-[:WORK_WITH]->(Neo4j)
• Coding
• Writing
• Speaking
• Helping
• Connecting
• Organizing

3. Topics
• Databases are No Fun
• Relational Pain -> Graph Fun
• The world is a Graph
• Neo4j
• Model, Query, Import
• Having Fun in the Developer Zone
• GitHub Events
• Software Analytics
• Neo4j from Java

4. Databases are No Fun
We have all been there

5. What pained me
• Object vs. Database Model = Pain
• Hard to Model
• Object relational impedance mismatch (and ORMs)
• Schema evolution = DBA Fights
• Slow queries = JOIN Pain
• Complex Queries = Pages of SQL
• Query Optimization = Denormalization
• Roundtrips (n+1 select, complex operations) = Cumbersome

6. What saved me? – Meeting Emil
Geek Cruise in 2008

7. Neo4j: A Story of Pain
With a Happy Ending

8. History of Neo4j - Problem
• Digital Asset Management System in 2000
• SaaS many users in many countries
• Two hard use-cases
• Multi language keyword search
• Including synonyms / word hierarchies
• Access Management to Assets for SaaS Scale
• Groups, Hierarchies, Permissions, Realtime

9. History of Neo4j – Relational Attempt
• Tried with many relational DBs
• JOIN Performance Problems
• Hierarchies, Networks, Graphs
• Modeling Problems
• Data Model evolution
• No Success, even …
• With expensive database consultants!

10. History of Neo4j – First working Implementation
• Graph Model & API sketched on a napkin
• Nodes connected by Relationships
• Just like your conceptual model
• Implemented network-database in memory
• Java API, fast Traversals
• Worked well, but …
• No persistence, No Transactions
• Long import / export time from relational storage

11. History of Neo4j - Solution
• Evolved to full fledged database in Java
• With persistence using files + memory mapping
• Transactions with Transaction Log (WAL)
• Lucene for fast entity lookup
• Founded Company in 2007
• Neo4j (REST)-Server
• Neo4j Clustering & HA
• Cypher Query Language
• Today …

12. Neo Technology Overview
Product
• Neo4j - World’s leading graph
database
• 1M+ downloads, adding 70k+
per month
• 150+ enterprise subscription
customers including over
50 of the Global 2000
Company
• Neo Technology, Creator of Neo4j
• 100+ employees with HQ in Silicon
Valley, London, Munich, Paris and
Malmö
• $45M in funding from Fidelity,
Sunstone, Conor, Creandum,
Dawn Capital

13. What, Who, Where, How?
Financial
Services
Communications
Health &
Life
Sciences
HR &
Recruiting
Media &
Publishing
Social
Web
Industry
& Logistics
Entertainment Consumer Retail Information ServicesBusiness Services
http://neo4j.com/use-cases http://neo4j.com/customers

14. Why should I care?
Because Relationships
Matter

15. What is it with Relationships?
• World is full of connected people, events, things
• There is “Value in Relationships” !
• What about Data Relationships?
• How do you store your object model?
• How do you explain
JOIN tables to your boss?

16. Neo4j – allows you to connect the dots
• Was built to efficiently
• store,
• query and
• manage highly connected data
• Transactional, ACID
• Real-time OLTP
• Open source
• Highly scalable already on few machines

17. Value from Data Relationships
Common Use Cases
Internal Applications
Master Data Management
Network and
IT Operations
Fraud Detection
Customer-Facing Applications
Real-Time Recommendations
Graph-Based Search
Identity and
Access Management

18. Neo4j Browser – Built-in Learning

19. RDBMS to Graph – Familiar Examples

20. Neo4j Browser – Visualization

21. Demo
Meetup Import

22. Teaser: Meetup.com Import
• For a Meetup Event
• Import Attendees
• For each Attendee
• Import Interests / Topics
• Import other Meetup Memberships
• Other groups our members are in
• Top 10 topics
• Topics & Groups of active Member
https://github.com/ikwattro/meetup2neo
http://markhneedham.com/blog?s=meetup

23. From RDBMS to Neo4j
Relational Pains =
Graph Pleasure

24. Relational DBs Can’t Handle Relationships Well
• Data Model built for tabular forms not JOINS
managing connections was bolted on both in
schema and query
• Strict schema not suitable for variable structured
data which is generated and used by todays
applications
• Data volume and JOIN number affect cost of query
operation exponentially
• Variable hierarchies and networks are hard to store
and query so many “patterns” were developed
… often only denormalization makes complex
relational queries fast but destroys the good
normalized data-model
Built for Forms
Joins are expensive
Denormalize #FTW

25. Unlocking Value from Your Data Relationships
• Model your data naturally as a graph
of data and relationships
• Drive graph model from domain and
use-cases
• Use relationship information in real-
time to transform your business
• Add new relationships on the fly to
adapt to your changing requirements

26. High Query Performance with a Native Graph DB
• Relationships are first class citizen
• No need for joins, just follow pre-
materialized relationships of nodes
• Query & Data-locality – navigate out
from your starting points
• Only load what’s needed
• Aggregate and project results as you
go
• Optimized disk and memory model
for graphs

27. Relational Versus Graph Models
Relational Model Graph Model
KNOWS
ANDREAS
TOBIAS
MICA
DELIA
Person FriendPerson-Friend
ANDREAS
DELIA
TOBIAS
MICA

28. For Instance …

29. … Is Actually
Order
Product
Customer Employee
SOLD
ORDERS
Category
Employee
REPORTS_TO
PART_OF
PURCHASED
Supplier
SUPPLIES

30. MATCH (boss)-[:MANAGES*0..3]->(mgr),
(mgr)-[:MANAGES*1..3]->(report)
WHERE boss.name = “John Doe”
RETURN mgr.name AS Subordinate,
count(report) AS Total
Express Complex Queries Easily with Cypher
Find all direct reports and how
many people they manage,
each up to 3 levels down
Cypher Query
SQL Query

31. High Query Performance: Some Numbers
• Traverse 2-4M+ relationships per
second and core
• Cost based query optimizer –
complex queries return in
milliseconds
• Import 100K-1M records per second
transactionally
• Bulk import tens of billions of records
in a few hours

32. Working with a Graph
Model, Import, Query

33. The Whiteboard Model Is the Physical Model
Eliminates Graph-to-
Relational Mapping
In your data model
Bridge the gap
between business
and IT models
In your application
Greatly reduce need
for application code

34. CAR
name: “Dan”
born: May 29, 1970
twitter: “@dan”
name: “Ann”
born: Dec 5, 1975
since:
Jan 10, 2011
brand: “Volvo”
model: “V70”
Property Graph Model Components
Nodes
• The objects in the graph
• Can have name-value properties
• Can be labeled
Relationships
• Relate nodes by type and direction
• Can have name-value properties
LOVES
LOVES
LIVES WITH
PERSON PERSON

35. Cypher: Powerful and Expressive Query Language
MATCH (:Person { name:“Dan”} ) -[:LOVES]-> (:Person { name:“Ann”} )
LOVES
Dan Ann
LABEL PROPERTY
NODE NODE
LABEL PROPERTY

36. Getting Data into Neo4j
Cypher-Based “LOAD CSV” Capability
• Transactional (ACID) writes
• Initial and incremental loads of up to
10 million nodes and relationships
Command-Line Bulk Loader neo4j-import
• For initial database population
• For loads up to 10B+ records
• Up to 1M records per second
4.58 million things
and their relationships…
Loads in 100 seconds!
CSV

37. Querying Your Data

38. Basic Pattern: Tom Hanks Movies?
MATCH (:Person {name:”Tom Hanks"} ) -[:ACTED_IN]-> (:Movie {title:”Forrest Gump"} )
ACTED_IN
Tom Hanks
Forrest
Gump
LABEL PROPERTY
NODE NODE
Forrest
Gump
LABEL PROPERTY

39. Basic Query: Tom Hanks‘ Movies?
MATCH
(actor:Person)-[:ACTED_IN]->(m:Movie)
WHERE actor.name = "Tom Hanks"
RETURN *

40. Basic Query: Tom Hanks‘ Movies?

41. Query Comparison: Colleagues of Tom Hanks?
SELECT *
FROM Person as actor
JOIN ActorMovie AS am1 ON (actor.id = am1.actor_id)
JOIN ActorMovie AS am2 ON (am1.movie_id = am2.movie_id)
JOIN Person AS coll ON (coll.id = am2.actor_id)
WHERE actor.name = "Tom Hanks“
MATCH
(actor:Person)-[:ACTED_IN]->()<-[:ACTED_IN]-(coll:Person)
WHERE actor.name = "Tom Hanks"
RETURN *

42. Basic Query Comparison: Colleagues of Tom Hanks?

43. Most prolific actors and their filmography?
MATCH (p:Person)-[:ACTED_IN]->(m:Movie)
RETURN p.name, count(*), collect(m.title) as movies
ORDER BY count(*) desc, p.name asc
LIMIT 10;

44. Most prolific actors and their filmography?

45. Neo4j Query Planner
Cost based Query Planner since Neo4j 2.2
• Uses database stats to select best plan
• Currently for Read Operations
• Query Plan Visualizer, finds
• Non optimal queries
• Cartesian Product
• Missing Indexes, Global Scans
• Typos
• Massive Fan-Out

46. Query Planner
Slight change, add an :Person label -> more stats
available -> new plan with fewer database-hits

47. Demo
Software Analytics
jqassistant.org

48. Software Analytics
Software is connected information, aka a graph
• Source -> AST
• Inheritance, Composition, Delegation
• Call Trees
• Runtime Memory
• Dependencies
• Modules, Libraries
• Tests
• ...
https://jqassistant.org

49. jQAssistant
• GeekCruise: My first Neo4j project
• Software deteriorates
• Develop rules and enforce them
• Commercial Tools too inflexible
• Open Source Software ...
• Scanner -> Enhancer -> Analyzer
• Enrichment, Concepts and Rulez in Cypher
• Scanner Plugins
• Integrate in Build Process
• Fail, Generate Reports, ... https://jqassistant.org

51. Let‘s explore ...
https://jqassistant.org
... The JDK

52. Demo
GitHub Events
github.com/ikwattro

53. Demo: GitHub Events
• GitHub: Social Coding
• GH-Events:
• Fork, Comment, PR, ...
• Archive + API
• Application for Import:
• GitHub ->PHP -> RabbitMQ -> Neo4j
• Data:
• 1M Users, 1.4M Repos, 62k Orgs
• 13M Events
https://github.com/ikwattro/github2cypher
https://github.com/ikwattro/github-event

54. Model: GitHub Events (partial)

55. Event: Watch a Repository
MATCH (w:WatchEvent)
WITH w LIMIT 1
MATCH p = (w)-[:EVENT_TIME]->(:Minute)<-[:CHILD*]-(:Year),
(w)-[:EVENT_ACTOR]->(u:User)-->(r:Repository)
<-[:WATCHED_REPOSITORY]-(w)
RETURN *;
https://twitter.com/ikwattro/status/618431227100532737

56. Using Neo4j from Java
Choices Galore!
Neo4j.com/developer/java

57. Neo4j: OSS, Made in Java
• Java 7 / 8, Scala
• High Performance IO, Memory Mapping,
• Collections, Caches, Cursors,
• Remoting,
• Paxos, Raft, ...
• Libraries
• Jetty, Lucene, Netty, Parboiled, ...
https://github.com/neo4j/neo4j

58. Using: Neo4j from Java
• Neo4j Java APIs
• Server Extensions
• Embedded
• (parallel) Batch-Import APIs
• REST / HTTP
• JDBC Driver
• Neo4j-OGM
• Spring Data Neo4j
• Upcoming: Binary Protocol Driver
https://neo4j.com/developer/java

59. Get up to speed with Neo4j
Quickly and Easily

60. There Are Lots of Ways to Easily Learn Neo4j
neo4j.com/developer

61. Thank You!
Ask Questions, or Tweet
@neo4j | http://neo4j.com
@mesirii | Michael Hunger