Boost Your Neo4j with User-Defined Procedures

© 2022 Neo4j, Inc. All rights reserved.
Michael Hunger,
Senior Director, User Innovation
@mesirii
Boost Your Neo4j Experience
with User-Defined Procedures

2
Outline
• Why should I care?
• Hello World Example
• UDP bridge the gap
• Java API vs. Cypher
• When should I use UDP? When should I build one?
• Development / Testing / Deployment
• Pro's and Cons
• Real Example
• Procedure Libraries
◦ APOC, GDS, neosemantics

3
Why should I care?
1 2
Efficiency Encapsulation
3 4
Integration Portability

4
Hello World Example
public class Hello {
@UserFunction
public String hello(@Name(„text“) String text) {
if (text == null) {
return null;
}
return String.format(„Hello %s!“, text);
}
}

User Defined Procedures & Functions
Neo4j Execution Engine
User Defined
Procedure
Applications
Bolt
User Defined Procedures & Functions let
you write custom code that is:
• Written in any JVM language
• Deployed to the Database
• Accessed by applications via Cypher
User Defined
Functions
5

Built-in procedures
SHOW PROCEDURES

Example: Listing all relationship types
CALL db.relationshipTypes()

Usage
CALL dbms.listQueries()
YIELD queryId, query, username, elapsedTimeMillis, activeLockCount
WHERE elapsedTimeMillis > 1000
CALL dbms.killQuery(queryId) yield message
RETURN queryId, substring(query,0,100) as query, username,
elapsedTimeMillis, activeLockCount, message

9
User Defined
● Procedures
● Functions
● Aggregation Functions
Bridge the Gap

10
Bridging the Gap
1 2
Exposed in
Cypher
Composable
Lego Blocks
3 4
Extend with
new
functionality
Power &
Ease

11
Procedures
● read / write
● stateless
● stream results
● standalone and
embedded
call proc(param)
yield value1,
value2
What is available? User Defined …
Functions
● read-only
● stateless
● single result
● in expressions
return func(param)
as value
Aggregation
Functions
● read-only
● stateful
● agg results
● in aggregations
return key,
agg(value) as
aggregated

12
When should I use user defined procedures & functions?
1 2
Functionality Performance
3 4
Compact-
ness
Missing
Cypher
Capability

13
Development

14
When should I create a user defined procedures?
1 2
Business
spec.need
Performance
SLA
3 4
Required
Integration
Missing
Cypher
Capability

15
Cypher vs. Java API
• original API
• low level
• imperative
• only available in database
extensions
• allows access to all Java data
and control structures
• additional concurrency
• easy to learn, write, maintain
• declarative
• graph patterns
• available through drivers &
tools
• limited data- and control-
structures
15

● any JVM language
● separate
procedure-layer
from
implementation
● avoid accessing
internal APIs
16
Develop
Development Cycle
Test
● can be unit tested
● Testcontainers for
integration tests
● IT also for
dependencies and
bolt integration
● biggest pain point
● version matches
● provide or
package
dependencies
● requires restart
● all cluster
members
Deploy

17
Pros and Cons
• More involved code to maintain
long-term (JVM skills)
• Deployment (server restart, aura,
cluster)
• No planner participation
• No resource mgmt participation
• No feedback about internal
operations
• Need to know what you are
doing
• Composable Lego blocks
• Encapsulate functionality
• Write once then reuse
• Most Java language features
• Testable with Unit testing
• Refactoring
• Apply permissions to
procedures
17

18
User Defined Function Example

User Defined Functions
● @UserFunction annotated, named Java Methods
○ default name: package + method
● take @Name'ed parameters (with default values)
● return a single value
● are read only
● can use @Context injected GraphDatabaseService and others
● run within transaction of the Cypher statement
19

20
Simple Function (Join)
public class Join {
@UserFunction
@Description("example.join(['s1','s2',...], delimiter) –
join the given strings with the given delimiter.")
public String join(
@Name("strings") List<String> strings,
@Name(value = "delimiter", defaultValue = ",") String delimiter) {
if (strings == null || delimiter == null) {
return null;
}
return String.join(delimiter, strings);
}
}

Calling the function from Cypher:
RETURN example.join(["Hello", "World", "Austin"], "-")
-> „Hello-World-Austin!“
Calling from Cypher
21

1. Deploy & Register in
Neo4j Server via
neo4j-harness
2. Call & test via
neo4j-java-
driver
Test Code
22

● Build or download (shadow) jar
● Drop jar-file into $NEO4J_HOME/plugins
● For sandboxed / allow-listing, need to edit neo4j.conf
dbms.security.procedures.unrestricted=example.*
● Restart server
● Functions & Procedures should be available (show procedures)
● Otherwise check neo4j.log / debug.log
Deploying User Defined Code
23

24
User Defined Procedure Example

User Defined Procedures
● @Procedure annotated, named Java Methods
● additional mode attribute (Read, Write, Dbms, Schema)
● return a Stream of value objects (DTO) with public fields
● value object fields are turned into columns
● Runs as part of outer transaction
● Injection of @Context fields, like Transaction, Log, Database
● Same testing & deployment as functions
github.com/neo4j-examples/neo4j-procedure-template
25

Procedure Code Example (Dijkstra)
@Procedure
@Description("apoc.algo.dijkstra(startNode, endNode, 'KNOWS', 'distance') YIELD path," +
" weight - run dijkstra with relationship property name as cost function")
public Stream<WeightedPathResult> dijkstra(
@Name("startNode") Node startNode,
@Name("endNode") Node endNode,
@Name("type") String type,
@Name("costProperty") String costProperty) {
PathFinder<WeightedPath> algo = GraphAlgoFactory.dijkstra(
PathExpanders.forType(RelationshipType.withName(type)),
costProperty);
Iterable<WeightedPath> allPaths = algo.findAllPaths(startNode, endNode);
return StreamSupport.stream(allPaths.spliterator(), false)
.map(WeightedPathResult::new);
}
26

28
Realworld Example
Optimizing Category/Genre
Overlap

29
Task: Category/Tag Overlap
Given a set of categories (names) find all items
that are in at least in these categories.
Simple for one, two or three categories:
MATCH (question)-[:TAGGED]->(:Tag {name:$name1}),
(question)-[:TAGGED]->(:Tag {name:$name2})
But not optimal, and not arbitrary.

Example Query - Tag Overlap
PROFILE
WITH ['cypher','neo4j', 'python'] AS tagNames
UNWIND tagNames AS name
MATCH (t:Tag {name:name})<-[:TAGGED]-(q:Question)
WITH q, collect(t) AS tags, tagNames
WHERE size(tags) = size(tagNames)
RETURN q
// 2.075.518 total db hits
30

Example Query - Tag Overlap
PROFILE
MATCH (t:Tag)<-[:TAGGED]-(q:Question)
WHERE t.name IN tagNames
WITH q, collect(t) AS tags, tagNames
WHERE size(tags) = size(tagNames)
RETURN q LIMIT 10
// 2.075.518 db-hits
31

Example Query - Tag Overlap - Optimized
PROFILE
MATCH (t:Tag {name:name})
// lowest degree first
WITH t ORDER BY size( (t)<-[:TAGGED]-() ) ASC
WITH collect(t) AS tags
WITH head(tags) AS first, tail(tags) AS rest
MATCH (first)<-[:TAGGED]-(q:Question)
WHERE all(t IN rest WHERE (q)-[:TAGGED]->(t))
RETURN q
// 89.137 total db hits
32

Example Query - Tag Overlap - Optimized (2)
PROFILE
MATCH (t:Tag {name:name})
// lowest degree first
WITH t ORDER BY size( (t)<-[:TAGGED]-() ) ASC
WITH collect(t) AS tags
WITH head(tags) AS first, tail(tags) AS rest
MATCH (first)<-[:TAGGED]-(q:Question)-[:TAGGED]->(other)
WITH q, collect(other) AS questionTags, rest
WHERE all(t IN rest WHERE t IN questionTags)
RETURN q
// 55.660 total db hits
33

Example - Tag Overlap - Procedure
1. pass in list of tag names
2. find tag with least questions (degree) as driver
3. put other tags into Set
4. iterate over questions of minimal tag
5. for each question
6. for each tag that is in set increment counter
7. if counter = Set size
a. add/stream question to result
34

Example - Genre Overlap - Procedure - Code (1)
// Procedure class with injection
public class SharedTags {
@Context public GraphDatabaseService gdb;
@Context public Log log;
enum Types implements RelationshipType {
TAGGED;
}
enum Labels implements Label {
Tag, Question
}
}
35

// Return Type, each field a column
public static class QuestionTag {
public String title;
public String tag;
public double weight;
public QuestionTag(String title,
String tag,
double weight) {
this.title = title;
this.tag = tag;
this.weight = weight;
}
}
36

@Procedure
public Stream<QuestionTag> sharedGenres(
@Name(”tagNames") List<String> tagNames) {
int tagCount = tagNames.size();
// lookup and order tags by degree ascending
List<Pair<Node, Integer>> tags =
tagNames.stream().map(tagName -> {
Node tag = gdb.findNode(Labels.Tag, "name", tagName);
return Pair.of(tag, tag.getDegree(Types.TAGGED,
Direction.INCOMING));
}).sorted((p1, p2) -> compare(p1.other(), p2.other())).toList();
37

// find questions for the tag with the lowest degree
var driver = tags.get(0);
var questions = new HashMap<>(driver.other());
for (Relationship e0 : driver.first()
.getRelationships(Direction.INCOMING, Types.TAGGED)) {
Node question = e0.getStartNode();
questions.put(question,
new ArrayList<>(List.of(Pair.of(driver.first(), e0))));
}
38

// check and add to found questions for remainder of tags
for (Pair<Node, Integer> tag : tags.subList(1, tags.size())) {
Node node = tag.first();
for (Relationship e1 : node.getRelationships(Direction.INCOMING,
Types.TAGGED)) {
Node question = e1.getStartNode();
if (questions.containsKey(question)) {
questions.get(question).add(Pair.of(tag.first(), e1));
}
}
}
39

// transform map into result stream
return questions.entrySet().stream()
.filter(entry -> entry.getValue().size() == tagCount)
.flatMap(entry -> {
String title = (String) entry.getKey().getProperty("title", null);
return entry.getValue().stream().map(tagRel -> {
String tagName = (String) tagRel.first().getProperty("name");
Number weight = (Number)tagRel.other().getProperty("weight",0);
return new QuestionTag(title, tagName, weight.doubleValue());
});
});
}
}
40

41
Existing Procedure Libraries

42
Existing Procedure Libraries
1 2
Built-in APOC
3 4
graph
data
science
neosemantics

43
Built In Procedures
Many built-in procedures are phased out for admin commands – Not a fan.
• Indexes & constraints esp. Fulltext (phased out)
• Schema visualization / data
• Cluster management
• Security / User Mmgt (phased out)
• Database state
• Procedures & Functions (phased out)
• Query, Transaction, Connection Management (phased out)
• Configuration Management
• Memory Pools
• Scheduler

44
Areas Of Use (APOC)

45
Graph Data Science

46
neosemantics

47
Learn more
1 2
Docs
neo4j.com/docs/java-reference
Guides
neo4j.com/developer/cypher/procedures-functions/
3 4
Example
github.com/neo4j-examples/
neo4j-procedure-template
Read the
source!

48
Q&A

49
Thank you!
Follow me: @mesirii

• Why do we need user defined procedures and functions?
• Syntax & built in procedures
• The APOC procedure library
• Helper functions
• Data integration
• Graph algorithms
• Graph refactoring / Transaction management
• How to write and test your own procedure / function
• Turn Cypher queries into procedures
What will we learn?

What are User Defined
Procedures and Functions?

Neo4j Developer Surface
Native LanguageDrivers
BOLT User Defined
Procedure
2000-2010 0.x - Embedded Java API
2010-2014 1.x - REST
2014-2015 2.x - Cypher over HTTP
2016 - 3.0.x - Bolt, Official Language Drivers, and User Defined Procedures
3.1.x - User Defined Functions

Using User Defined Procedures

Using - User Defined Procedures
http://neo4j.com/docs/developer-manual/current/procedures
// shortcut for stand alone call
CALL dbms.procedures()
// process result columns
YIELD name, signature, description
RETURN count(name)

Exercise: List available procedures
1. List all procedures in the "dbms" namespace
2. List all procedures whose signature contains "NODE"
3. Group procedures by first part of the names (e.g. "db") and return a
count of the number of procedures and a collection containing just
the last part of the name. (e.g. "awaitIndex")
4. Play around with at least 3 procedures in the list. Which ones did you
try?

Listing procedures
WITH * WHERE name STARTS WITH "dbms."
RETURN *

Listing procedures
WITH * WHERE signature CONTAINS "NODE"
RETURN *

Listing procedures
WITH split(name,".") AS parts
RETURN parts[0] AS package,
count(*) AS count,
collect(parts[-1]) AS names

Cons of User-Defined Procedures
• Quite involved CALL clause for simple, read-only computation,
conversion functions or predicates
• Need to YIELD and select result columns
• Can’t be part of expressions

User-Defined Functions
• Allows users to create their own functions and use them with Cypher
• Useful for expressing common computations, rules, conversions,
predicates
• Functions can be used in any expression, predicates
• Extend the Neo4j 3.0 Stored Procedure mechanism

Creating a UUID with a Procedure vs. with a Function
CALL apoc.create.uuid() YIELD uuid
CALL apoc.data.formatDefault(timestamp(), "ms")
YIELD value AS date
CREATE (:Document {id: uuid, created:date})
vs
CREATE (:Document {
id: apoc.create.uuid(),
date: apoc.data.formatDefault(timestamp(), "ms")})

APOC
Awesome Procedures on Cypher
Why and How?

Download the appropriate version of APOC and put it
in the $NEO4J_HOME/plugins directory
https://github.com/neo4j-contrib/neo4j-apoc-
procedures#version-compatibility-matrix
After you’ve done this restart Neo4j
Install APOC

APOC Procedures
• Cypher is expressive and great for graph operations but misses some
utilities so people built their own one-off solutions
• APOC is a "Standard Library" of helpful procedures and functions
• Started as an experiment but has evolved into an active community
project with 500+ procedures and 75+ functions
• github.com/neo4j-contrib/neo4j-apoc-procedures

What does APOC cover?
• Functions for date, time, data conversion, collection handling
• Procedures for data integration, graph algorithms, graph refactoring,
metadata, Cypher batching
• TimeToLive (TTL), triggers, parallelization
• And much more!

Documentation
• documentation site github.com/neo4j-contrib/neo4j-apoc-procedures
• browser guide :play http://guides.neo4j.com/apoc
• many articles & blog posts neo4j.com/tag/apoc

User-Defined Functions in APOC - Packages

(Meta)-Utilities, Converters

(Meta)-Utilities, Converters
• apoc.date.format(timestamp(),"ms","YYYY-MM-dd")
• apoc.number.parse("12.000,00")

The Meta Graph

Use dbms.functions() to:
1. List all functions in the "apoc" namespace
2. Play around with at least 3 other functions. Which ones did you try?
Exercise: Explore APOC functions

RETURN apoc.coll.toSet([1,2,3,4,4,4,4,5,6,7])
APOC functions

RETURN apoc.text.clean(" Neo4j ")
APOC functions

Data Import and Export

Data Integration / Import
• Databases
• JDBC
• MongoDB
• Cassandra
• Elastic
• CouchDB
• File formats
• JSON
• XML

LOAD JDBC
Load from relational database, either a full table or a sql statement
CALL apoc.load.jdbc('jdbc-url','TABLE') YIELD row
CALL apoc.load.jdbc('jdbc-url','SQL-STATEMENT') YIELD row
To simplify the JDBC URL syntax and protect credentials, you can
configure aliases in conf/neo4j.conf:
apoc.jdbc.alias.url=jdbc:mysql://localhost:3306/<database>?user=<username>
CALL apoc.load.jdbc('alias','TABLE')

Load Northwind data from MySQL

Load Northwind data: Products and Orders

Load Northwind data: Order Details

Load Northwind data: Customers

// import XML as single nested map with attributes and `_type`, `_text` fields
// and `_<childtype>` collections per child-element-type.
CALL apoc.load.xmlSimple('xml-url')
YIELD value as doc
Loading data from XML

Load Northwind data: Graph Result

Product Recommendations

// load from JSON URL (e.g. web-api) to import JSON as stream of values if the JSON
was an array or a single value if it was a map
CALL apoc.load.json('json-url')
YIELD value as row
Loading data from JSON

// load CSV fom URL as stream of values
CALL apoc.load.csv('csv-url',{config})
YIELD lineNo, list, map
// config contains any of:
{skip: 1,
limit: 5,
header: false,
sep: 'TAB',
ignore:['tmp'],
arraySep: ';',
mapping: {
years: { type:'int', arraySep:'-', array:false, name:'age', ignore:false}
}
Loading data from CSV

Exercise: Import StackOverflow
Use APOC's apoc.load.json to import StackOverflow from this URI:
https://api.stackexchange.com/2.2/questions?pagesize=1
00&tagged=neo4j&site=stackoverflow
You can also find it in the documentation:
• neo4j-contrib.github.io/neo4j-apoc-procedures
• Search for "Load JSON StackOverflow Example"

Simple Example:
Write, Test, Use a Function

● @UserFunction annotated, named Java Methods
○ default name: package + method
● take @Name'ed parameters (with default values)
● return a single value
● are read only
● can use @Context injected GraphDatabaseService
● run within transaction of the Cypher statement

@UserFunction("create.uuid")
@Description("creates an UUID (universally unique id)")
public String uuid() {
return UUID.randomUUID().toString();
}
Simple Function (UUID)

@UserFunction("create.uuid")
@Description("creates an UUID (universally unique id)")
public String uuid() {
return UUID.randomUUID().toString();
}
Calling the function from Cypher:
RETURN create.uuid();
CREATE (p:Person {id: create.uuid(), name:{name}})
Simple Function (UUID)

1. Deploy & Register in Neo4j Server via neo4j-harness
2. Call & test via neo4j-java-driver
Test Code
https://github.com/neo4j-contrib/training/

1. Deploy & Register in Neo4j Server via neo4j-harness
2. Call & test via neo4j-java-driver
@Rule
public Neo4jRule neo4j = new Neo4jRule().withFunction( UUIDs.class );
try( Driver driver = GraphDatabase.driver( neo4j.boltURI() , config ) {
Session session = driver.session();
String uuid = session.run("RETURN create.uuid() AS uuid")
.single().get( 0 ).asString();
assertThat( uuid,....);
}
Test Code

● Build or download (shadow) jar
● Drop jar-file into $NEO4J_HOME/plugins
● Restart server
● Functions & Procedures should be available
● Otherwise check neo4j.log / debug.log
Deploying User Defined Code

● @Procedure annotated, named Java Methods
● additional mode attribute (Read, Write, Dbms)
● return a Stream of value objects with public fields
● value object fields are turned into columns
User Defined Procedures
https://github.com/neo4j-examples/neo4j-procedure-template

Procedure Code Example (Dijkstra)
@Procedure
@Description("apoc.algo.dijkstra(startNode, endNode, 'KNOWS', 'distance') YIELD path,"
+
" weight - run dijkstra with relationship property name as cost function")
public Stream<WeightedPathResult> dijkstra(
@Name("startNode") Node startNode,
@Name("endNode") Node endNode,
@Name("type") String type,
@Name("costProperty") String costProperty) {
PathFinder<WeightedPath> algo = GraphAlgoFactory.dijkstra(
PathExpanders.forType(RelationshipType.withName(type)),
costProperty);
Iterable<WeightedPath> allPaths = algo.findAllPaths(startNode, endNode);
return Iterables.asCollection(allPaths).stream()
.map(WeightedPathResult::new);
}

@Rule
public Neo4jRule neo4j = new Neo4jRule()
.withProcedure( Dijkstra.class );
try( Driver driver = GraphDatabase.driver( neo4j.boltURI() , config ) {
Session session = driver.session();
String query =
"MATCH ... CALL apoc.algo.dijkstra(...) YIELD path RETURN ...";
String pathNames = session.run(query)
.single().get( 0 ).asString();
assertThat( pathNames, ....);
}
Test Procedure

Periodic Execution &
Transaction Control

Large operations? Consider TX batching
CALL apoc.periodic.iterate('
CALL apoc.load.jdbc(
"jdbc:mysql://localhost:3306/northwind?user=root", "company")',
'CREATE (p:Person) SET p += value',
{batchSize:10000, parallel:true})
RETURN batches, total

Exercise 2: Import 5 pages of StackOverflow
Use APOC's apoc.periodic.iterate with the apoc.load.json
solution you wrote earlier.
You can also find information about in the documentation:
neo4j-contrib.github.io/neo4j-apoc-procedures

Graph Algorithms
101

Graph Algorithms
• While these are neat, favor the Graph Algo procedures instead
• https://neo4j.com/docs/graph-algorithms/current/
• Pathfinding: apoc.algo.dijkstra, apoc.algo.aStar
• PageRank: apoc.algo.pageRankStats, apoc.algo.pageRankWithCypher
• Centrality: apoc.algo.betweenness, apoc.algo.closeness
• Clustering: apoc.algo.community, apoc.algo.wcc, apoc.algo.cliques
• Node-Cover: apoc.algo.cover

103
Split 3rds - bullets
When using bullets on the left ,
select all the bullets and use:
Add space after list item
to visually separate them
Example of bullets:
• Short point
here not more
than 3 lines
• Another point
which is not too
long
• Short and
sweet

104
Neo4j Aura Enterprise is allowing us to realize our
vision to build a knowledge graph that unifies all
product knowledge to improve our retail
recommendations, search and merchandising.”
“
- Name and position, Company

105
Special slides examples
1 2
Arial
Extra Bold 86pt
for big numbers
Arial
Normal 14pt for
small text
3 4
Arial
Extra Bold 86pt
for big numbers
Arial
Normal 14pt for
small text

106
Thing to compare 2
Thing to compare 1
• Comparison 1
• Comparison 2
◦ Details
• Comparison 1
• Comparison 2
◦ Details

107
1. Use Media Slide from Master
2. Image is cropped at width
4.62in, so it takes up half of
the slide (and aligned with
blue rectangle on the left)
Media

108
Image Placeholder

109

110
Your Logo Here

111
111
Your Logo Here

Boost Your Neo4j with User-Defined Procedures

More Related Content

What's hot

Similar to Boost Your Neo4j with User-Defined Procedures

More from Neo4j

Recently uploaded

In this document

Boost Your Neo4j with User-Defined Procedures