Neo4j - London User Group Meetup - 28th March, 2018 If your data ingestion requirements have grown beyond importing occasional CSV files then this talk is for you. Neo4j-Databridge from GraphAware is a comprehensive ETL tool specifically built for Neo4j. It has been designed for usability, expressive power and high performance to address the most common isues faced when importing data into Neo4j - multiple data sources and type, very large data sets, bespoke data conversions, non-tabular formats, filtering, merging and de-duplication, as well as bulk imports and incremental updates. In this talk, we'll take a quick tour of the some of the main features, loading data from Kafka, Redis, JDBC and various other data sources along the way, to understand how Neo4j Databridge solves these problems and how it can help you import your data quickly and easily into Neo4j. Vince Bickers is a Principal Consultant at GraphAware and the main author of Spring Data Neo4j (v4). He has been writing software and leading software development teams for over 30 years at organisations like Vodafone, Deutsche Bank, HSBC, Network Rail, UBS, VMWare, ConocoPhillips, Aviva and British Gas.

1. GraphAware®
Neo4j-Databridge
Enterprise-ready ETL for Neo4j
from GraphAware

2. Introductions
What is Neo4j-Databridge?
Who is it for?
Overview: Resources, Schema mappings and Adapters
Road map
Demo!
Agenda
GraphAware®

3. You’ve outgrown LOAD CSV
You’re doing continuous ingestion
You want to do “What if” snapshotting
You are doing data aggregation into Neo4j
You have complex and/or time consuming imports
Who is it for?
GraphAware®

4. Easy to use and configure
Fast and scalable
Strategies for merging and managing duplicates
Supports multiple data formats and data providers
Automatic conversion from JDBC -> Graph
Expressions for filtering and data conversions
Supports full and incremental imports
Offline and online endpoints
User-extensible via simple APIs
Can integrate easily into an existing Neo4j deployment
Core features
GraphAware®

5. High-level view
GraphAware®
Databridge
CSV
Adapter
JDBC
Adapter
Kafka
Adapter
Excel
Adapter

6. Overview
GraphAware®
Resource Definitions
Schema Mappings
• A Databridge import Task consists of one or more user-
defined specifications, defined using JSON:
• The Databridge import engine uses built-in Adapters to
perform the import
• No need to pre-process data beforehand
• No need to know Cypher
• No need to write code
• If you can write a JSON file, you’re good to go.

7. Resource Definitions
GraphAware®
Define a data resource
Connection info
Configuration properties
Metadata (e.g delimiters)

8. Files
GraphAware®
{
"resource": "~/csv-files/*.csv"
}
{
"resource": "ftp://alice@jabberwocky/data/geo-data.json"
}
{
"resource": "excel-data/sales-2017-q1.xls"
}
Excel
JSON
CSV
File resources for common file formats.

9. JDBC
GraphAware®
{
"resource": "jdbc:mysql://localhost/employees?useSSL=false",
"query": "SELECT * from departments"
}
{
"resource": "jdbc:mysql://localhost/employees?useSSL=false",
"query": "departments-query.sql"
}
Import from any JDBC-compliant store

10. Message-processing
GraphAware®
{
"resource": "kafka://localhost:9092/messages",
"timeout": "none"
}
{
"resource": "kafka://localhost:9092/messages?partition=2",
“topic.offset": 0,
"timeout": 30000
}
Import from Kafka
• for as long as Kafka is running
• or until a user-specified timeout period has elapsed
• define seek point for rewinding if required

11. Key-Value/Cache
GraphAware®
{
"adapter": "com.graphaware.databridge.adapters.redis.RedissonAdapter",
"nodeAddresses": "redis://localhost:6379, … “,
"redisQueueName": “test-queue”,
..
}
Import from Redis
{
"resource": "redis://localhost:6379/test-queue"
}

12. Core Adapters
GraphAware®
Text
Excel
JDBC
Redis
Kafka
Core adapters provided “out of the box”

13. Data formats
GraphAware®
CSV
JSON
GPX
ITN
CIF
PGN
Support for tabular and non-tabular data formats

14. Custom Adapters
GraphAware®
• Extend AbstractAdapter
• Compile and deploy to /lib folder
• Declare in Resource Definition
Adapter API allows you to build and deploy your own adapters
{
"name" : "my-custom-resource",
"adapter" : "com.mycompany.MyCustomAdapter",
"resource" : "resources/custom.resource"
}

15. Schema mappings
GraphAware®
A schema mapping
• References the resource to be imported
• Describes Node and Edge mappings
{
"resource" : "messages-resource.json",
"nodes": [
{
"type" : "Phone.1",
"identity": ["Src"]
},
{
"type" : "Phone.2",
"identity": ["Tgt"]
}
],
"edges" : [
{"type":"CALLED", "source": "Phone.1", "target": "Phone.2",
"properties": [
{"name": "at", "column": "Sent"}
]
}
]
}

16. Update strategies
GraphAware®
{
"resource" : "messages-resource.json",
"nodes": [
{
"type" : "Phone.1",
"identity": ["Src"],
"update_strategy":"unique"
},
{
"type" : "Phone.2",
"identity": ["Tgt"],
"update_strategy":"unique"
}
],
"edges" : [
{"type":"CALLED", "source": "Phone.1", "target": "Phone.2",
"properties": [
{"name": "at", "column": "Sent"}
],
"update_strategy":"version"
}
]
}
• Unique
• Merge
• Version
• automatically selected if not defined

17. Expressions
GraphAware®
{
"resource" : "messages-resource.json",
"nodes": [
{
"type" : "Phone.1",
"identity": ["Src"],
"update_strategy":"unique"
},
{
"type" : "Phone.2",
"identity": ["Tgt"],
"update_strategy":"unique"
}
],
"edges" : [
{"type":"CALLED", "source": "Phone.1", "target": "Phone.2",
"properties": [
{"name": "at", "column": “Sent”, “condition”: "at > 0"}
],
"update_strategy":"version"
}
]
}
• Conditionally create nodes, edges, properties and labels
• Supports sub-expressions (…), and/or/not, RegEx and all the usual suspects.

18. Data converters and formatters
GraphAware®
{
"resource" : "messages-resource.json",
"nodes": [
{
"type" : "Phone.1",
"identity": ["Src"],
"update_strategy":"unique"
},
{
"type" : "Phone.2",
"identity": ["Tgt"],
"update_strategy":"unique"
}
],
"edges" : [
{"type":"CALLED", "source": "Phone.1", "target": "Phone.2",
"properties": [
{"name": "at", "column": “Sent”, "convert": "iso8601-timestamp"}
],
"update_strategy":"version"
}
]
}
• Dates
• Numbers
• Strings

19. Custom data conversions
GraphAware®
• Write your own data converters in Groovy and drop them into the plugins folder
package plugins
import com.graphaware.neo4j.databridge.plugins.*
map = [
'LEO': 'Low-Earth Orbit',
'MEO': 'Mid-Earth Orbit',
'HEO': 'High-Earth Orbit',
'L1': 'Lissajous 1',
'L2': 'Lissajous 2'
]
def bind() {
{ args ->
map.get(args[0])
} as Converter
}
this
• Then use them in your schema mappings
{
"nodes": [
{
"type": "Location",
"properties": [ { "name": “location", "column": "Alt",
"convert": "plugins.orbit_location" } ],
"identity": [ "location" ]
}
]
}

20. Road map
GraphAware®
More adapters (JMS, Elastic, …)
Developer version
Integration with the Neo4j Graph Platform
Web-based UI (under development)
email: databridge@graphaware.com
ask for an evaluation copy
Want to know more?

21. csv - load NASA satellite data from a CSV file
excel - load UK Charity data from an Excel spreadsheet
jdbc - auto-import a MySql database
itn - load the Isle of Man TT route into Neo4j
gpx - import all UK Aviation waypoints (navaid)
kafka - real-time RSS news feeds into Neo4j
redis - :-(
hawkeye - load 15m graph objects in < 1 minute
Demo time!
GraphAware®