Extending Gremlin with Foundational Steps

Extending Gremlin with
Foundational Steps
2019

Introduction
A Short Explanation of Graphs and Apache TinkerPop

© DataStax, All Rights Reserved.ConfidentialConfidential © DataStax, All Rights Reserved.
What is a Graph?
● A property graph is a collection of vertices and edges
○ A vertex represents entities/domain objects
○ An edge represents a directional relationship between two vertices
○ Vertices and edges have labels and properties
label: person
name: Stephen
label: organization
name: DataStax
label: person
name: Paras
label: employs
since: 2015
label: employs
since: 2016
label: knows

Apache TinkerPop
● An open source graph computing
framework for both graph databases
(OLTP) and graph analytic systems
(OLAP)
● Provides the Gremlin graph traversal
language
● Exposes an agnostic way by which users
can build graph applications
● Wide industry support to include DataStax
Graph (currently, there are over two dozen
graph systems are TinkerPop-enabled)

Gremlin
The Graph Traversal Language of Apache TinkerPop

Gremlin is a
functional, data-flow
language that enables
users to succinctly
express complex
traversals on their
application's property
graph.
“Who does Stephen know?”
g.V().has("person","name","stephen").
out("knows").
values("name")

This query
demonstrates the
three hallmarks of
novice Gremlin usage:
1. Filtering
2. Navigation
3. Simple
Transformation
out("knows").
values("name")

What skill
tends to
separate the
novice from
the advanced
Gremlin user?

The advanced
Gremlin user
can easily
extract data
that they need
from a traversal
in the form that
they want.

Novice to Advanced
A Thin Line Between Novice and Advanced Gremlin with a Customer-360 Style Use Case

Expanded Example Graph
organizationperson
employs
knows
event
hosts
presents
attends hosts

Expanded Example Graph
g.addV('person').property('name','stephen').property('timeZone','EST').as('s').
addV('person').property('name','paras').property('timeZone','EST').as('p').
addV('organization').property('name','datastax').
property('city','santa clara').property('state','CA').as('ds').
addV('event').property('name','DataStax Accelerate').
property('presentation','Extending Gremlin with Foundational Steps').
property('year','2019').property('attendees',100).as('da').
addV('event').property('name','Cassandra & DataStax DC Meetup').
property('presentation','Gremlin Queries with DataStax Enterprise Graph').
property('year','2017').property('attendees',35).as('dm0').
addE('knows').from('s').to('p').
addE('employs').from('ds').to('p').property('since',2016).
addE('employs').from('ds').to('s').property('since',2015).
addE('attends').from('s').to('da').
addE('presents').from('s').to('da').
addE('hosts').from('ds').to('da').
addE('attends').from('s').to('dm0').
addE('presents').from('s').to('dm0').
addE('hosts').from('p').to('dm0').iterate()

Consider these ego-
centric traversals that
use purely novice-
level Gremlin (i.e.
filtering, navigation,
simple
transformations) to
gather information
from the local
subgraph about
“stephen”.
out("knows").
valueMap("name","timeZone")
“Who has employed Stephen?”
inE("employs").
order().by("since",desc).
outV().
valueMap("name","city","state")
“How many events did Stephen attend?”
outE("attends").
count()

out("knows").
inE("employs").
outV().
outE("attends").
count()
JSON
[{
"name": ["paras"],
"timeZone":
["EST"]
}]
JSON
[{
"city": ["santa
clara"],
"name": ["datastax"],
"state": ["CA"]
}]
JSON [2]

{
"colleagues": [{
"name": "paras",
"timeZone": "EST"
}],
"employers": [{
"city": "santa clara",
"name": "datastax",
"state": "CA"
}],
"eventsAttended": 2
}
What does an
application need in terms
of these individual
results?
1. A single traversal
instead of multiple
traversals
2. Simplified collections -
flattened property
values rather than
lists

Steps for Collection Construction
● Lists
○ store()
○ aggregate()
○ fold()
○ path()
○ union()
● Maps
○ group()/groupCount()
○ project()
○ select()
○ valueMap()/propertyMap()
Collection Manipulation
● unfold()
● range()
● limit()
● tail()

out("knows").
inE("employs").
outV().
outE("attends").
count()
{
"colleagues": [{
"name": "paras",
"timeZone": "EST"
}],
"employers": [{
"city": "santa
clara",
"name": "datastax",
"state": "CA"
}],
"eventsAttended": 2
}
Note that...
The traversals have the same
starting filter and thus begin with
the same vertex. Each traverses
away from that vertex to gather
data about “stephen”

out("knows").
inE("employs").
outV().
outE("attends").
count()
{
"colleagues": [{
"name": "paras",
"timeZone": "EST"
}],
"employers": [{
"city": "santa
clara",
"name": "datastax",
"state": "CA"
}],
"eventsAttended": 2
}
Note that...
The desired result is a Map with
user defined keys each holding a
result that contain data gathered
by traversing away from
“stephen”.

● unfold()
● range()
● limit()
● tail()
● Lists
○ store()
○ aggregate()
○ fold()
○ path()
○ union()
● Maps
○ project()
○ select()
Use project() when the Map
keys are known/static and
when features of the current
traverser are meant to be
represented for each key.

gremlin> g.V().has("person","name","stephen")
==>v[0]
gremlin> g.V().has("person","name","stephen").
......1> project("colleagues", "employers", "eventsAttended")
==>[colleagues:v[0],employers:v[0],eventsAttended:v[0]]
......1> project("colleagues", "employers", "eventsAttended").
......2> by(fold())
==>[colleagues:[v[0]],employers:[v[0]],eventsAttended:[v[0]]]
Use of project()
project() accepts a by()
modulator for each specified
key which is responsible for
the transformation to apply to
v[0] for that key.
1
2
3

......1> outE("attends").
......2> count()
==>2
......2> by(fold()).
......3> by(fold()).
......4> by(outE("attends").count())
==>[colleagues:[v[0]],employers:[v[0]],eventsAttended:2]
Use of project() - “eventsAttended” key
Projects v[0] to
“eventsAttended” key using
the “How many events did
Stephen attend?” traversal.
1
2

......1> out("knows").
......2> valueMap("name","timeZone")
==>[name:[paras],timeZone:[EST]]
......1> out("knows").
......2> valueMap("name","timeZone").
......3> by(unfold())
==>[name:paras,timeZone:EST]
Use of project() - “colleagues” key
unfold() deconstructs a
List to a stream and by()
only grabs the first object of
the child traversal. The effect
is to unwrap the embedded
List.
1
2

......1> out("knows").
......2> valueMap("name","timeZone").
......3> by(unfold())
==>[name:paras,timeZone:EST]
......2> by(out("knows").valueMap("name","timeZone").by(unfold()).fold()).
......3> by(fold()).
......4> by(outE("attends").count())
==>[colleagues:[[name:paras,timeZone:EST]],employers:[v[0]],eventsAttended:2]
Use of project() - “colleagues” key
Projects v[0] to “colleagues”
key using the “Who does
Stephen know?” traversal.
1
2

......1> inE("employs").
......2> order().by("since",desc).
......3> outV().
......4> valueMap("name","city","state").
......5> by(unfold())
==>[city:santa clara,name:datastax,state:CA]
......1> project("colleagues","employers","eventsAttended").
......2> by(out("knows").valueMap("name","timeZone").by(unfold()).fold()).
......3> by(inE("employs").
......4> order().by("since",desc).
......5> outV().
......6> valueMap("name","city","state").
......7> by(unfold()).
......8> fold()).
......9> by(outE("attends").count()).next()
==>colleagues=[{name=paras, timeZone=EST}]
==>employers=[{city=santa clara, name=datastax, state=CA}]
==>eventsAttended=2]
Use of project() - “employers” key
Projects v[0] to “employers”
key using the “Who has
employed Stephen?” traversal.
1
2

project("colleagues","employers","eventsAttended").
by(out("knows").
valueMap("name","timeZone").
by(unfold()).
fold()).
by(inE("employs").
outV().
valueMap("name","city","state").
by(unfold()).
fold()).
by(outE("attends").count())
{
"colleagues": [{
"name": "paras",
"timeZone": "EST"
}],
"employers": [{
"city": "santa
clara",
"name": "datastax",
"state": "CA"
}],
"eventsAttended": 2
}

Gremlin DSLs
Building Domain Specific Languages (DSLs) with Gremlin

A Gremlin-
based DSL
allows
traversals to be
written in the
language of the
application
domain.

Benefits of Gremlin DSLs
● Hide traversal complexity
● Improve Gremlin testability
● Improve code readability
● Improve code maintainability and reusability

Gremlin DSL
concepts can
apply to an
application
domain, but also
to fundamental
extension of the
graph domain.

{
"colleagues": [{
"name": "paras",
"timeZone": "EST"
}],
"employers": [{
"city": "santa
clara",
"name": "datastax",
"state": "CA"
}],
"eventsAttended": 2
}
by(out("knows").
valueMap("name","timeZone").
by(unfold()).
fold()).
by(inE("employs").
outV().
valueMap("name","city","state").
by(unfold()).
fold()).

● Lists
○ store()
○ aggregate()
○ fold()
○ path()
○ union()
● Maps
○ project()
○ select()
○ simpleMap()
● unfold()
● range()
● limit()
● tail()
This looks like
a helpful step!

@GremlinDsl
● TinkerPop provides a Java annotation processor in gremlin-
core to help build DSLs.
● The @GremlinDsl annotation is applied to an interface that
extends the GraphTraversal interface with new steps.
● The annotation processor then generates the appropriate
boilerplate code and includes the new step.
● While this topic is discussed in Java, DSLs can be built in any
Gremlin Language Variant.

@GremlinDsl - simpleMap()
@GremlinDsl()
public interface FoundationTraversalDsl<S,E> extends GraphTraversal.Admin<S,E> {
default GraphTraversal<S,Map<Object,Object>> simpleMap(String... keys) {
return valueMap(keys).by(__.unfold());
}
}

@GremlinDsl - Usage
DseSession session = DseSession.builder().build();
RemoteConnection conn = DseGraph.remoteConnectionBuilder(session).build();
FoundationTraversalSource g = traversal(FoundationTraversalSource.class).
withRemote(conn);
Map<String,Object> m = g.V().has("person","name","stephen").simpleMap().next();

{
"colleagues": [{
"name": "paras",
"timeZone": "EST"
}],
"employers": [{
"city": "santa
clara",
"name": "datastax",
"state": "CA"
}],
"eventsAttended": 2
}
by(out("knows").
simpleMap("name","timeZone").
fold()).
by(inE("employs").
outV().
simpleMap("name","city","state").
fold()).

“What events has Stephen attended and who were the presenters/hosts?”
by(out("knows").
fold()).
by(inE("employs").
outV().
fold()).
by(out("attends").
order().by("attendees",desc).
map(union(simpleMap("name","presentation"),
inE("presents","hosts").
group().
by(label).
by(outV().values("name"))).
unfold().
group().by(keys).by(select(values))).
fold())
{
"colleagues": [{
"name": "paras",
"timeZone": "EST"
}],
"employers": [{
"name": "datastax",
"state": "CA"
}],
"eventsAttended": [{
"presentation": "Extending Gremlin...",
"hosts": "datastax",
"name": "DataStax Accelerate",
"presents": "stephen"
}, {
"presentation": "Gremlin Queries...",
"hosts": "paras",
"name": "Cassandra & DataStax DC
Meetup",
}]
}

by(out("knows").
fold()).
by(inE("employs").
outV().
fold()).
by(out("attends").
group().
by(label).
unfold().
fold())
{
"colleagues": [{
"name": "paras",
"timeZone": "EST"
}],
"employers": [{
"name": "datastax",
"state": "CA"
}],
}, {
"hosts": "paras",
Meetup",
}]
}
out("attends").
group().
by(label).
unfold().
fold()

by(out("knows").
fold()).
by(inE("employs").
outV().
fold()).
by(out("attends").
group().
by(label).
unfold().
fold())
{
"colleagues": [{
"name": "paras",
"timeZone": "EST"
}],
"employers": [{
"name": "datastax",
"state": "CA"
}],
}, {
"hosts": "paras",
Meetup",
}]
}
out("attends").
group().
by(label).
unfold().
fold()
out("attends").
simpleMap("name","presentation")
out("attends").
group().
by(label).
by(outV().values("name"))
1
1
2 2

by(out("knows").
fold()).
by(inE("employs").
outV().
fold()).
by(out("attends").
group().
by(label).
unfold().
fold())
{
"colleagues": [{
"name": "paras",
"timeZone": "EST"
}],
"employers": [{
"name": "datastax",
"state": "CA"
}],
}, {
"hosts": "paras",
Meetup",
}]
}
out("attends").
group().
by(label).
unfold().
fold()
Merging
two Maps!

● Lists
○ store()
○ aggregate()
○ fold()
○ path()
○ union()
● Maps
○ project()
○ select()
○ singleMap()
● unfold()
● range()
● limit()
● tail()
● merge()
● partition()
● interleave()
● zip()
These look like
helpful steps!

@GremlinDsl - merge()
@GremlinDsl()
}
default GraphTraversal<S, Map> merge(Traversal... maps) {
return map(__.union(maps).
unfold().
group().
by(keys).
by(__.select(values)));
}
}

by(out("knows").
fold()).
by(inE("employs").
outV().
fold()).
by(out("attends").
merge(simpleMap("name","presentation"),
group().
by(label).
fold())
{
"colleagues": [{
"name": "paras",
"timeZone": "EST"
}],
"employers": [{
"name": "datastax",
"state": "CA"
}],
}, {
"hosts": "paras",
Meetup",
}]
}
out("attends").
group().
by(label).
fold()

by(out("knows").
fold()).
by(inE("employs").
outV().
fold()).
by(out("attends").
group().
by(label).
fold())
{
"colleagues": [{
"name": "paras",
"timeZone": "EST"
}],
"employers": [{
"name": "datastax",
"state": "CA"
}],
}, {
"hosts": "paras",
Meetup",
}]
}

@GremlinDsl - partition()
@GremlinDsl()
}
...
default GraphTraversal<S, List<Object>> partition(int size) {
return emit().
until(__.not(__.unfold())).
repeat(__.skip(local,size)).
filter(__.unfold()).
limit(local,size);
}
}

gremlin> g.inject([1,2,3,4,5,6])
==>[1,2,3,4,5,6]
gremlin> g.inject([1,2,3,4,5,6]).partition(2)
==>[1,2]
==>[3,4]
==>[5,6]
gremlin> g.inject([1,2,3,4,5,6]).partition(3)
==>[1,2,3]
==>[4,5,6]
Use of partition()
1
2
3

@GremlinDsl - interleave()
@GremlinDsl()
}
...
default GraphTraversal<S,Map<Object,Object>> interleave() {
return index().
unfold().
order().
by(__.tail(local)).
limit(local,1);
}

gremlin> g.inject([1,2,3],['a','b','c'],[7,8,9]).interleave()
==>1
==>a
==>7
==>2
==>b
==>8
==>3
==>c
==>9
gremlin> g.inject([1,2,3],['a','b','c'],[7,8,9]).interleave().fold().partition(3)
==>[1,a,7]
==>[2,b,8]
==>[3,c,9]
Use of interleave()
1
2

@GremlinDsl - zip()
@GremlinDsl()
}
...
default GraphTraversal<S,Map<Object,Object>> zip() {
return ((FoundationTraversal) interleave()).
fold().
partition(2).
group().
by(__.limit(local,1)).
by(__.skip(local,1).unfold());
}
}
Build more complex DSL
steps based on other
lower level DSL steps.

gremlin> g.inject(['a','b','c'],[4,5,6])
==>[a,b,c]
==>[4,5,6]
gremlin> g.inject(['a','b','c'],[4,5,6]).zip()
==>[a:4,b:5,c:6]
Use of zip()
1
2

gremlin> g.inject(['a','b','c'],[4,5,6]).zip()
==>[a:4,b:5,c:6]
gremlin> g.inject(['a','b','c'],[4,5,6]).
......1> index().
......2> unfold().
......3> order().
......4> by(tail(local)).
......5> limit(local,1).fold().
......6> emit().
......7> until(__.not(unfold())).
......8> repeat(skip(local,2)).
......9> filter(unfold()).
.....10> limit(local,2).
.....11> group().
.....12> by(limit(local,1)).
.....13> by(skip(local,1).unfold())
==>[a:4,b:5,c:6]
Use of zip()
1
2

Gremlin DSL
Resources
1. DSL Introduction (Java
oriented) -
https://s.apache.org/bipK
2. DSL design guidelines
(Python oriented) -
https://s.apache.org/1xiv
3. DSL pattern for “enrichment”
(.NET oriented) -
https://s.apache.org/knOS
4. DSL sample code -
https://s.apache.org/UNiz

Extending Gremlin with Foundational Steps

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Extending Gremlin with Foundational Steps

Similar to Extending Gremlin with Foundational Steps (20)

Recently uploaded

Recently uploaded (20)

Extending Gremlin with Foundational Steps