Graph Database Query Languages

Graph Database Query Languages
Jay Coskey
2018-09-05
slideshare.net/jaycoskey
1

2
• Databases, Graphs, and Graphviz
• Property Graphs & Cypher
• TinkerPop & Gremlin
• Knowledge Graphs/Triples
• Conclusions
Graph Query Languages

Databases, Graphs, and Graphviz
3

Types of Databases
4
Commercial examples Open Source examples
Relational DBs Oracle, MSFT SQL Server MySQL/Maria, Postgres
Column family DBs (think CSVs) Apache Cassandra, Apache HBase
Key-value stores AWS DynamoDB, Oracle Berkeley DB Redis
Document DBs Apache Couchbase, MongoDB
Graph DB — Property Graph
(for traversing relations in data)
neo4j, DataStax Enterprise Graph,
Amazon Neptune, Memgraph
JanusGraph (from Titan), Dgraph
Graph DB — RDF
(for finding relations in data)
Stardog, onotext, AllegroGraph blazegraph
Graph DB — Multi-Model
(e.g., doc store w/ graph layer)
ArangoDB, ElasticSearch, OrientDB, SAP,
MSFT CosmosDB, Virtuoso Universal Server
Graph processing frameworks
(Apache)
TinkerPop & Gremlin
Spark GraphX
Giraph & Hama (Hadoop) & S2Graph
Graph processing frameworks
(Non-Apache)
Oracle, ThingSpan, InfoGrid
GraphBase, Trovares xGT
Apache Hama

Graphs
Graph = 2-tuple: <V, E>
• V = set of vertices
• E = set of edges
(possibly with properties)
Directed:
V = {A, B, C, D, E, F}
E = {
(A, E), (D, B),
(B, C), (C, F), (F, E), (E, B)
}
A B C
D E F
• Weighted graph: A graph with values (weights) associated with edges
• Example: A graph with a cost for each edge traversal
• Multigraph: Vertex pairs can be connected by multiple edges
• Example: Multiple emails can be sent from personA to personB
• See Wikipedia's Glossary of graph theory terms for much more.
A B
D E
w=0.75 w=2.1
A B
D E
5

Visualizing Graphs with Graphviz (dot)
digraph G { // Layout not right
A -> E[color=red,label=0.75];
D -> B[color=red,label=2.1];
B -> C[color=red];
C -> F[color=red];
F -> E[color=red];
E -> B[color=red];
A [style=filled, color="#a0a0ff"];
B [style=filled, color="#a0a0ff"];
C [style=filled, color="#a0a0ff"];
D [style=filled, color="#a0a0ff"];
E [style=filled, color="#a0a0ff"];
F [style=filled, color="#a0a0ff"];
}
digraph G {
rankdir=LR;
// Same content as at left
subgraph left { rankdir=TB; rank=same; A; D; };
subgraph center { rankdir=TB; rank=same; B; E; };
subgraph right { rankdir=TB; rank=same; C; F; };
}
% dot –Tpng mygraph.dot > mygraph.png # dot file format below with output
6

Visualizing Graphs with Graphviz (pydot)
from pydot import Dot, Edge, Node, Subgraph
import random
import string
def main():
vdict = {label: Node(label, style='filled', fillcolor='#a0a0ff')
for label in list('ABCDEF')
}
verts = vdict.values()
graph = Dot(graph_type='digraph', rankdir='LR')
def add_edge(src, dst, label='', color='red'):
graph.add_edge(Edge(vdict[src], vdict[dst], label=label, color=color))
def add_subgraph(label_chars):
sg_name = ''.join(random.choice(string.ascii_lowercase) for _ in range(5))
sg = Subgraph(sg_name, rankdir='TB', rank='same')
for v in get_verts(list(label_chars)):
sg.add_node(v)
graph.add_subgraph(sg)
def get_verts(label_chars):
return [vdict[label] for label in label_chars]
7
for v in verts:
graph.add_node(v)
add_edge('A', 'E', label=0.75); add_edge('D', 'B', label=2.1)
add_edge('B', 'C'); add_edge('C', 'F')
add_edge('F', 'E'); add_edge('E', 'B')
add_subgraph('AD')
add_subgraph('BE')
add_subgraph('CF')
graph.write_dot(mygraph.dot')
graph.write_png(mygraph.png') // Can output multiple formats

Property Graphs
10
• Property Graphs have (many?) properties associated with vertexes and/or edges.
• Example: Family tree metadata: supporting (conflicting?) documents, uncertainty, etc.
First words
Adoption details
Inventions
Nickname
Earings, necklace
Suitors
Catchphrases Photos
Monolithic representation
vs.
Property-based
mix & match link & layer
Simpsons image from englishexercises.org

Some Property Graph Domains
11
Domain Sample Entities & Properties Sample Application
Airline travel Airplane, Flight Attendant, Departure, Engineer, Passenger, Pilot, Route Optimize staff assignments
Case law Case, Defendant, Jurisdiction, Law, Prosecutor, Ruling, Trial, Verdict Choose which cases to cite
Criminal Investigation Person, Email, Jurisdiction, PhoneCall, TimeSpan, University, Workplace Identify suspicious activity
Genealogy Person, Adoption, Birth, Burial, Child, Death, Marriage, Parent, Spouse Family tree match
Language origins Word, Document, IsReconstructed, Language, Pronunciation, UsageDate Linguistic phylogenetics
Medicine Patient, Death, Diagnosis, Doctor, Drug, Measurement, Treatment Finding patient zero
Movies Movie, Actor, Director, Language, Length, Producer, Rating, Review, Year Casting; Distance to Kevin Bacon
Social networks User, Ad, AdCampaign, AdPlacement, DateTime, Group, Message, Post Recommend friends
Supply chain mgmt Product, Carrier, Cost, LocDst, LocSrc, Quantity, Shipment, Warehouse Optimize ordering & delivery
System Engineering Component, FailureRate, Inputs, RecoveryTime, OutageCost, Outputs Root cause determination

Modeling Graphs in SQL: It can be done ...
WITH subordinates (id, name) AS (
SELECT id, name FROM org_chart AS _base
WHERE boss_name = 'Alice'
)
UNION ALL
(
SELECT id, name FROM org_chart AS _next, subordinates AS subs
WHERE _next.boss_id = subs.id
)
SELECT id, name FROM subordinates;
12
Query adapted from Joe Celko's Trees and Hierarchies in SQL for Smarties
recursion
org_chart
id
name
boss_id
boss_name

Modeling Graphs in SQL: ... but sometimes it shouldn't
SQL supports indexes to improve query performance:
But ...
• Native Graph DBs instead use storage, processing, and indexing optimized for graphs.
• They "pre-materialize" (i.e., index/cache) relationships to improve performance.
13
Small memory footprint Few page loads Efficient query
Index to search only
the columns needed
Clustered indexes to search
fewer blocks
Long path queries
have many joins
Exploding number
of paths
Large memory
footprint
Domain Edges/Vertex
Org Charts A handful
Emails 100s or 1000s
Network traffic Millions

Example of Neo4j's Cypher Syntax (MATCH)
14
MATCH ( movie: Movie { title: "The Matrix" } )
RETURN movie
1
This label is not necessary, but can
greatly improve performance.
The parentheses here are required.
They signify a vertex/node.

More Examples of Cypher Syntax (MATCH)
15
MATCH (user)->[:FRIEND]->(friend)
WHERE user.name = "Amy"
WITH user, count(friend) AS num_friends
WHERE num_friends >= 10
RETURN user // Users named Amy w/ 10+ friends
MATCH (u)->[:FRIEND]->(f1)->[:FRIEND]->(foaf),
(u)->[:FRIEND]->(f2)->[:FRIEND]->(foaf)
WHERE f1 <> f2
RETURN u, foaf // Friends of friends by 2+ connections
topological constraints
Before WITH After WITH
user, friend user, num_friends
f1
f2
foafu
f1 f10
Amy
2
3
vertex vertexedge
v e v e v

Refresher on SQL SELECT Syntax
As written:
SELECT ...
[ FROM ... [JOIN ... ] ]
[ WHERE ...]
[ GROUP BY ... ]
[ HAVING ... ]
[ ORDER BY ... ]
[ WINDOW ... ]
16
Order of execution:
FROM
JOIN
OUTER
WHERE -- Pre-filter
GROUP BY -- Aggregation
HAVING -- Post-filter
SELECT
DISTINCT
ORDER BY
LIMIT/TOP
-- Result
-- Source
(Lots of syntactic details left out)

Comparison: SQL SELECT vs Cypher's MATCH Syntax
SQL:
SELECT cols
[ FROM ... [JOIN ... ] ]
[ WHERE ...]
[ GROUP BY ... ]
[ HAVING ... ]
[ ORDER BY ... ]
[ WINDOW ... ]
17
Cypher:
[MATCH paths WHERE ...]
[OPTIONAL MATCH paths WHERE ...]
[WITH ... [ORDER BY ...] [SKIP ...] [LIMIT ...]]
RETURN exprs
[ORDER BY ...] [SKIP ...] [LIMIT ...]
MATCH clause: Names, types, and topology
RETURN clause: What is returned to the caller.
The SELECT clause specifies
what is returned to the caller,
and what those values are called.
topological constraints

Cypher: MATCH Syntax Details
• Specify patterns of paths: (vertex) and ->[edge]-> and <-[edge]<-
• Allow for variable length paths: ->[edge*2..]->
• Allow specification of type: (v:Employee), [e:ReportsTo]
• Allow specification of property values: (v: {name: 'Amy', isActive: true})
• Many things work (mostly) like in SQL:
• dates, math/string functions, NULL, UNION, CASE clauses, MERGE, user-defined stuff,, etc.
• Some differences:
18
UNWIND converts a list to rows collect() converts rows to a list
UNWIND [1, 2, 3, NULL] AS x
RETURN x, 'val' AS y // Returns 4 rows
// Earlier, we would have used FOREACH
WITH [1, 1, 2, 2] AS coll
UNWIND coll AS x
WITH DISTINCT x
RETURN collect(x) AS setOfVals

Examples of a Variable Length Paths (MATCH)
19
// Note: Path vars can be assigned in MATCH, CREATE, & MERGE statements
MATCH p = shortestPath(
(emma:Person {name:"Emma Watson"})-[*]-(bacon:Person {name:"Kevin Bacon"})
)
RETURN p // The path from Emma Watson to Kevin Bacon
MATCH (a: { name: 'Amy' })<-[e:ReportsTo:*2..]<-(emp)
WHERE a.isActive = true AND emp.isActive = true
RETURN emp.empId, emp.firstName, emp.lastName // Indirect reports of Amy
4
5

Property Graphs: Performance Issues
20
Factors that can greatly impact graph query performance:
• Size/Complexity. Paths lengths & count of connected components
• Memory management — The number of page accesses
a. The WITH statement promotes memory efficiency
b. Indexes can reduce the number of memory accesses
• Concurrency — Multiple tasks queued up simultaneously
• Parallelism — Multiple tasks executing simultaneously
• E.g.: Vector processing executes identical instructions simultaneously
Note: Cartesian products between disconnected patterns
cause warnings.

Cypher: Performance Optimization
21
Optimization Examples
Note: Cypher has an EXPLAIN statement.
Note: Some MATCHES benefit from choosing a START node.
Add INDEX hints after a MATCH clause:
USING INDEX variable:Label(property)
USING INDEX SEEK variable:Label(property)
// Multiple USING clauses can be used.
MATCH ... liskov:Scientist {name: 'Liskov'} ...
USING INDEX liskov:Scientist(name)
RETURN liskov.born AS birthdate
Add SCAN hints after a MATCH clause:
USING SCAN variable:Label
MATCH { s:Scientist }
USING SCAN s:Scientist
WHERE s.born < 1939
RETURN s.born AS birthdate
Add USING JOIN ON to provide JOIN hints. USING JOIN ON liskov
USING PERIODIC COMMIT with LOAD CSV commits the
current transaction periodically to minimize memory use, at
the cost of breaking transactional isolation.
USING PERIODIC COMMIT 500
By default, Neo4j uses a cost-based planner
with a statistics service. The user can set config
dbms.cypher.planner to RULE, & prepend queries
with CYPHER planner = rule

Graph Frameworks
• If you don't need the persistence & multi-user concurrency of databases:
• Apache Spark's GraphX is popular for data pipelines, and supports distributed queries.
val graph = GraphLoader.edgeListFile(sc, "data/graphx/followers.txt")
val ranks = graph.pageRank(0.0001).vertices
val users = sc.textFile("data/graphx/users.txt").map { line =>
val fields = line.split(",")
(fields(0).toLong, fields(1))
}
val ranksByUsername = users.join(ranks).map { case (id, (username, rank)) => (username, rank) }
• (Parallel) Boost Graph Library (BGL) is a parallel version of the classic C++ BGL.
typedef adjacency_list<listS, vecS, directedS, no_property, property<edge_weight_t, int>> Graph;
typedef graph_traits<Graph>::vertex_descriptor Vertex;
typedef std::pair<int,int> Edge; // Graph algorithms similarly templatized
• NetworkX is a Python graph library.
nx.draw(flights, pos=airport2latlong, alpha=0.75)
• Seattle's Trovares has a property graph engine called xGT, focused on
distributed performance. It might someday also feature a database.
• We'll focus on a framework called TinkerPop.
23
GraphX query from PageRankExample.scala in Apache Spark's github repo

TinkerPop & Gremlin
• Apache TinkerPop is an interface for computing with
(OLTP) graph DBs & (OLAP) graph analytical systems.
• Write once  Run on many DB engines.
• It comes with lots of tools/features:
• Platform-agnostic: Gremlin console & server, TinkerGraph
• Platform-specific: SparkGraphComputer & Hadoop support
• Supporting systems:
• Blazegraph, ChronoGraph, DataStax Enterprise Graph,
• JanusGraph, Neo4j, OrientDB, Stardog, etc.
• Languages with Gremlin APIs:
• Clojure, Groovy, Java, .NET, Python, Scala
• SPARQL & SQL can both be compiled to Gremlin
24
TinkerGraph diagram

Gremlin Examples
• Gremlin is a functional, data-flow (i.e., fluent) graph traversal language. Think LINQ.
• g.v("Amy").outE('friend').inV().age; // Ages of all Amy's friends
• g.V().interval("age", 25, 35); // People ages 25-35 (interval)
• g.V().filter{ it.age >= 25 }.filter{ it.age <= 35 }; // People ages 25-35 (lambdas)
• g.V().and(_().has('age', T.ge, 25), _().has('age', T.le, 35)); // People ages 25-35
• g.V().hasLabel("person").
pageRank().
by("friendRank").
by(outE("knows"))
order().by("friendRank", desc).
limit(10) // 10 highest-friend-ranked people
25
Plus, Gremlin has a REPL!! (Called the console)
The TinkerPop family:

Knowledge Graphs
27
• The drive for the semantic web led to Triplestores, or Resource Description Framework (RDF) stores.
• Graphs reduced to a canonically simple representation, using additional nodes instead of properties.
• Standardized "vocabularies" provide nuanced meanings (e.g., indep vs. dep continuants & occurrents)
• SQL provides syntactially uniform schemas to a domain. RDF provides flexible semantic schemas.
• Each triple is of the form <Subject, Predicate, Object> (Often these are URLs, for uniqueness.)
• Each triple represents a single edge.
• Examples (from Contacts): Person.name, age, birthdate
• Examples (from Geography): Capitol, City, Continent, Country
• An ontology is a standard set of triples. A knowledge graph is a dataset conforming to a set of ontologies.
• Ontologies or "vocabularies" can be imported via their prefix and reference URL.
Africa
Kenya
Senegal
EgyptCairo
Dakar
Nairobi
isCapital
isInContinent

Knowledge Graph Examples
28
etiological process
produces
disorder
bears
disposition
pathological
process
realized_in
abnormal
bodily
features
diagnosis interpretive
process /
tests
used_in
signs &
symptoms recognized_asproduces
Pathology
& Diagnosis
People
& Skills
Pathology image from Basic Formal Ontology: A Common Standard, by Barry Smith; skills diagram adapted from computer.org
Pizza & Toppings

SPARQL (RDF Query Language) Queries
29
SELECT (COUNT(?item) AS ?count) # List all persons on Wikidata
WHERE { ?item wdt:P31 wd:Q5 . } # P31 = is-instance-of. Q5 = Human. Result = 4,502,374 (Pedro Aguirre Cerda, etc.)
SELECT (COUNT(?item) AS ?count) # List all categories of persons on Wikidata
WHERE { ?item wdt:P279+ wd:Q5 . } # P279 = is-category-of. Result = 812 (child, man, woman, Asian, astronaut, author, etc.)
SELECT * WHERE { # List person-mother couples in the Terminator universe
?jc wdt:P1080 wd:Q620588 . # P1080 = is-in-fictional-universe; Q620588 = Terminator
?jc rdfs:label ?jc_lbl .
FILTER (langMatches(lang(?jc_lbl), "en"))
?jc wdt:P25 ?sc . # ?sc is-mother-of ?jc
?sc rdfs:label ?sc_lbl .
FILTER (lang(?sc_lbl) = "en") # Result
}
PREFIX geo: <http://www.geography.org/schema.rdf> # Find African countries/capitals
SELECT ?country ?capital # The wildcard character * would return these names as well as ?x & ?y.
WHERE {
?x:City geo:cityname ?capital ; # ?x has name ?capital
geo:isCapitalOf ?y . # Subject x is implied
?y:Country geo:countryname ?country ; # y has name ?country
geo:isInContinent geo:Africa . # y is in Africa
} # Sample query from Wikipedia
Prefix Meaning
wdt Wikidata type
rdfs RDF Schema
jc jc_lbl sc sc_lbl
wd:Q376577 John Connor wd:Q496437 Sarah Connor
1
2
3
4
Semi-colon: Subject carries over to next statement
Comma: Subject & predicate carry over

Regex-like RDF Path Expressions
30
Syntax Meaning
path+, path*, path? Specified range of copies
path{n,m}, path{n}, path{n,}, path{,m} Specified range of copies
path1|path2 Alternation: a choice of paths
path1/path2 A sequence of paths
^path1 Reversal of path (obj -> subj instead of subj -> obj)
SELECT (COUNT(?item) AS ?count)
WHERE { ?item wdt:P279+ wd:Q5 . } # One or more category-of-person tags

Some Ontologies
31
Prefix Description URI
rdf Resource Description Framework http://www.w3.org/1999/02/22-rdf-syntax-ns#
rdfs RDF Schema http://www.w3.org/2000/01/rdf-schema#
bfo Basic Formal Ontology http://www.ifomis.org/bfo/1.1
bio Biographical information http://purl.org/vocab/bio/01
dc Dublin Core http://purl.org/dc/elements/1.1
foaf Friend of a Friend http://xmlns.com/foaf/0.1
geo Latitude, longitude, altitude http://www.w3.org/2003/01/geo/wgs84_pos#
gfo General Formal Ontology http://www.onto-med.de/ontologies/gfo.owl#
gr Good Relations: Products Sold Online http://purl.org/goodrelations/v1
owl Web Ontology Language http://www.w3.org/2002/07/owl#
owl-time Temporal references (Date + time) http://www.w3.org/2006/time#
prov Provenance http://www.w3.org/ns/prov#
qo Quality measurements http://purl.org/qual/qual-o
sem Simple Event Model http://semanticweb.cs.vu.nl/2009/11/sem/
sioc Socially Interconnected Online Communities http://rdfs.org/sioc/spec
ssn Semantic Sensor Network (Sensors & Observations) http://purl.oclc.org/NET/ssnx/ssn
trn Transit http://vocab.org/transit
xsd XML Schema http://www.w3.org/2001/XMLSchema#

SPARQL: Syntax (SELECT)
32
BASE ... // Provide lexical prefixes for URLs
PREFIX ... // Reference vocabularies, such as rdf, rdfs, owl, xsd, dc, foaf
SELECT ...
FROM ...
FROM NAMED ... // The default dataset is the MERGE of all the FROM graphs
WHERE { ... FILTER ... GRAPH ... OPTIONAL ... SERVICE ... UNION ... }
GROUP BY ...
HAVING ...
ORDER BY ...
LIMIT ...
OFFSET ...
BINDINGS VALUES ...
Other SPARQL queries besides SELECT:
• ASK Ask whether there are matches: true/false
• CONSTRUCT Build RDF triples/graphs
• DESCRIBE Return RDF triples describing the given entity
Some non-query commands: CLEAR, CREATE, DELETE, DROP, INSERT, LOAD
Define temporary graphs
Communicate with remote endpoint, using SPARQ protocol
SPARQL's version of OUTER joins

RDF Collections
33
A subject or object can be a collection (cons-style lists):
( "apple" "banana" )
is shorthand for
[ rdf:first "apple";
rdf:rest [ rdf:first "banana";
rdf:rest rdf:nil
]
]
RDF collections can be nested.

n-Ary Relationships
35
• In knowledge graphs, n-ary relationships can be represented as nodes that
represent multi-entity relationships.
"Brutus stabbed Caesar with a knife at the Theater of Pompey on Mar 15, 44 BCE. "
Brutus Caesar
stabs
{ weapon: 'knife',
where: 'Theater of Pompey',
date: 'Mar 15, 44 BCE' }
Brutus CaesarStabbing
isSubject isObject
Knife
weapon where
44 BCE
when
Theater
Property Graph Knowledge Graph

Choosing Between Graph Databases
36
• In general, native graph DBs provide better the best graph performance.
• Property graph DBs are better for navigating relationships.
• E.g., Supply chain DB
• Knowledge graph DBs are better at discovering relationships.
• E.g., Criminal activity DB

Resources: Online SPARQL REPLs
38
Name URL Description
DBPedia http://dbpedia.org/sparql Data from Wikipedia
bio2rdf http://bio2rdf.org/sparql Bioinformatics data from ~40 public DBs
Wikidata https://query.wikidata.org Wikidata
Ontotext http://data.ontotext.com/sparql Ontotext demo

Resources: Other
Property Graphs
• Cypher refcard: https://neo4j.com/docs/cypher-refcard/current/
• Cypher Improvement Proposals (CIPs): https://www.opencypher.org/cips
• Efficient graph algorithms in Neo4j: https://neo4j.com/blog/efficient-graph-algorithms-neo4j/
TinkerPop & Gremlin
• Overview of Gremlin: http://tinkerpop.apache.org/gremlin.html
• The Gremlin Compendium, by Doan DuyHai: http://www.doanduyhai.com/blog/?p=13460
• List of platforms supporting TinkerPop: http://tinkerpop.apache.org/providers.html
Knowledge Graphs
• Defining N-ary relationships on the Semantic Web: https://www.w3.org/TR/swbp-n-aryRelations/
• SPARQL FAQ: http://www.thefigtrees.net/lee/sw/sparql-faq
• SPARQL RDF Reference v1.8: https://www.dajobe.org/2005/04-sparql/SPARQLreference-1.8.pdf
• W3C docs:
• RDF datasets: https://www.w3.org/wiki/DataSetRDFDumps
• RDF concepts: https://www.w3.org/TR/rdf11-concepts/
• SPARQL Extensions: https://www.w3.org/wiki/SPARQL/Extensions
• SPARQL Overview: https://www.w3.org/TR/sparql11-overview/
• SPARQL Protocol: https://www.w3.org/TR/sparql11-protocol/
• Turtle: Terse RDF Triple Language: https://www.w3.org/TeamSubmission/turtle/
• Wikibooks: https://en.wikibooks.org/wiki/SPARQL
39

Graph Database Query Languages

More Related Content

What's hot

Similar to Graph Database Query Languages

More from Jay Coskey

Recently uploaded

Graph Database Query Languages

Editor's Notes