This document provides an introduction to graph databases. It defines a graph store as a tool for storing and retrieving highly related data where many things are connected to many other things. It notes that graph databases are optimized for this type of data and discusses some popular graph database implementations. It then explores why graph databases may be useful and some limitations. The document provides examples of graph data modeling and querying capabilities. It also outlines some advanced graph database features and how to interact with a graph database using different programming languages.

1. Introduction to
Graph Databases
For Melbourne Data Engineering
by Timothy Findlay

2. What is a graph store ?
– Tool for storing and retrieving data
– Optimized for highly related data; where many things are connected to many other things
– There are many implementations such as Neo4J, Dgraph, ArangoDB, OrientDB, Titan/JanusGraph, DSE
Why use a graph ?
• High performance queries at scale
• Many-to-many relationships
Why not ?
• Slow range scans / initial seek
• Fixed depth, short scans
• Super Nodes eg. Everything is connected to everything
Stuart
Bob
Kevin

3. Data model problems
Relational databases are great at this, but can
struggle at scale with high number of joins
Some application databases are not designed for
analytical workloads which require joins
Some warehouses are not designed to provide
easy access to a variety of information without
slow and complex joining
Person Address
Credit Card
Address
Credit Card
Person
Vertex’s can labelled to form virtual layers
to partition data
eg.
Find address for each IP
Find buildings affected by a Cable
10.1.2.3
Layer: TCP/IP
Desk: WRK12:05
Layer: Building
10.1.2.4
Layer: TCP/IP
Cable 00154672
Layer: Physical
08:00:27:3d:90:82
Layer: Physical
08:00:27:73:4b:89
Layer: Physical
Desk: WRK12:05
Layer: Building
CableDuct: W775-12
Layer: Building
MobileMobile
Credit Card
Examples of traversing relationships
1 Someplace Ave
Layer: Address
1-2 Someplace Ave
Layer: Address
2 Someplace Ave
Layer: Address

4. Directionality
House House House
Bricks
Suburb: Kensington
Wood Asbestos A material may exist in a house and a houses may exist in a
suburb, but you may never know both sides of the equation
unless you search by house and traverse outward
For directed graphs the direction is a key design consideration
and needs to be considered carefully up front
Train Station

5. Fancy features
– Vertex revisions
Computer
Person
Some graph databases support versioning of vertexes.
This can used as a form of version control for audit, or to see what traversals
were like at a different point in time.
It could also be used to compare points in time to understand changes in
traversals eg. changes to the route of a network, plumbing, transport system
Person
Person
Computer
• Event Sourcing & Triggers
• Consider integrating Flume/Flink/Nifi pipelines into a graph
• Event sourcing can be a powerful way of establishing edges on the fly from a Kafka topic or JMS message
Company
Event1 Event2 Event3 Event4
If Company
has Event2
+ Event 4
Then …

6. Variations between implementations
– Storage layers – RocksDB, Cassandra, BerkleyDB, in-memory
– Types of query languages – Gremlin, GraphQL+, openCypher
– Different cluster technologies (Partitioning/Sharding)
– Enforced Schema and Schema-less support
– Different data types (document stores vs key/value pairs)
– Geospatial support
– Support for database functions
– Record versioning / effective dating
RDBMS / SQL GRAPH / GRAPHQL
database database
table collection
row document / vertex
column attribute
table joins collection joins (graph call these edges)
primary key primary key (automatically present on _key attribute)
index index
Terminology

7. How to use a graph store ?
– Starting
– docker run -p 8529:8529 arangodb/arangodb:3.0.10
– Putting data in
– echo { "name" : "Timothy" } | curl --basic --user “root:openSesame" -X POST --data-binary @- --dump -
http://localhost:8529/_api/gharial/MyPeople/vertex/people
– Pulling data out
– curl --basic --user “root:openSesame" -X GET http://arangodb:8529/_api/document/people/timothyfindlay/
– curl -X POST --dump - http://arangodb:8529/_api/people/
{ "query" : "FOR x in people FILTER IS_IN_POLYGON( x.loc, [ 153.090667 , -27.247108 ] , true ) == true RETURN
[ x.name ]", "bindVars": { "id" : 3 } }
This presentation is NOT sponsored by Arango GMBH

8. Python(eg. Airflow operator)
from pyArango.connection import *
import names
conn = Connection(arangoURL="http://10.1.20.6:8529", username="root", password="openSesame")
conn.createDatabase(name="MelbourneDEM")
db = conn["MelbourneDEM"]
table = db.createCollection(name="people")
for i in xrange(1000):
doc = table.createDocument()
doc["_key"] = names.get_full_name()
doc.save
CLI(eg. Cron batch processing)
arangoimp --on-duplicate ignore
--log.level warn
--server.endpoint http+tcp://10.1.20.6:8529
--server.authentication 'true'
--server.username root
--server.password openSesame
--server.database MelbourneDEM
--type csv
--create-collection true
--create-collection-type edge
--file edges.csv
--collection people_links

9. Java eg. Flume connector, Kafka Connect Sink, Nifi processor
<dependency>
<groupId>com.arangodb</groupId>
<artifactId>arangodb-java-driver</artifactId>
<version>4.1.12</version>
</dependency>
ArangoDB a = new ArangoDB.Builder()
.host(arangoDBHost, arangoDBPort)
.user(arangoDBUser)
.password(arangoDBPass)
.maxConnections(maxConnections)
.registerModule(new VPackDriverModule())
.build();
ArangoDatabase b = a.createDatabase(dbName);
ArangoCollection c = b.createCollection(collectionName);
MultiDocumentEntity d = c.insertDocuments( .... )

10. Javascript eg. NodeJS, Browser applications
'use strict';
// Constants
const PORT = 8888;
db = require('arangojs')();
db.useDatabase('pni');
// App
const app = express();
app.get('/', function (req, res) {
db.query('FOR x in people FILTER IS_IN_POLYGON( x.loc, [ 153.090667 , -27.247108 ] , true ) == true RETURN [ x.name ]').then(
cursor => cursor.all()
).then(
keys => res.send('All keys:', keys.join('<BR/> ')),
err => console.error('Failed to execute query:', err)
);
});
app.listen(PORT);

11. Whats next…
– Query languages like GraphQL
– Clustering and scaling
– More ingestion pipelines eg. Nifi, Kafka Connect
– Embedded functions / events