SlideShare a Scribd company logo
(Perl)-[:speaks]->(Neo4j)
Mark A. Jensen
1
https://github.com/majensen/rest-neo4p.git
• Perler since 2000
• CPAN contributor (MAJENSEN) since 2009
• BioPerl Core Developer
• Director, Genomic Data Programs,
Leidos Biomedical Research Inc (FNLCR)
• @thinkinator, LinkedIn
2
Not my sponsor, but could be yours!
• http://www.perlfoundation.org/how_to_write_a_proposal
3
Motivation
• Cancer Genomics:
Biospecimen, Clinical, Analysis Data
– complex
– growing
– evolving technologies
– evolving policies
– need for precise accounting
• Graph models are well-suited to this world
4
5
Patient
Tumor
Sample
Clinical
Extract
Extract
Data
File
Data
File
Normal
Sample
derived_from analysis_of
• age
• diagnosis
• stage
• date shipped
Nodes
Relationships
Properties
Graph vs RDBMS
6
foo
barbaz
spam
eggs
squirrel
goob
7
select bar.name
from bar, bar_baz, baz, baz_goob, goob,
goob_squirrel, squirrel, squirrel_spam, spam, spam_eggs,
eggs, eggs_foo, foo
where bar.id = bar_baz.bar_id and bar_baz.baz_id = baz.id and
baz.id = baz_goob.baz_id and baz_goob.goob_id = goob.id and
goob.id = goob_squirrel.goob_id and goob_squirrel.id = squirrel.id
and
squirrel.id = squirrel_spam.squirrel_id and
squirrel_spam.spam_id = spam.id and spam.id = spam_eggs.spam_id and
spam_eggs.eggs_id = eggs.id and eggs_foo.eggs_id = eggs.id and
eggs_foo.foo_id = foo.id and foo.name = 'zloty';
match (f:foo)-[*5..8]-(b:bar) where f.name = 'zloty' return b.name
Neo4j
• “Native” graph DB engine (currently in v2.2)
– Written in Java, but
– Very complete REST API
– Custom query language: Cypher
– Free community edition
– Lots of community support, including many
“language drivers”
• Not the only one out there, but probably the
most widely used (certainly the best
marketed)
8
Neo4p
9
Neo4p
10
Neo4p
11
Create Node
Label Node
Create
Unique Node
Add a Prop
Link Nodes
Load/Use
Index
Neo4p
12
Design Goals
• "OGM" – Perl 5 objects backed by the graph
• User should never have to deal with a REST endpoint*
*Unless she wants to.
• User should never/only have to deal with Cypher
queries†
†Unless he wants/doesn’t want to.
• Robust enough for production code
– System should approach complete coverage of the REST
service
– System should be robust to REST API changes and server
backward-compatible (or at least version-aware)
• Take advantage of the self-describing features of the API
13
REST::Neo4p core objects
• Are Node, Relationship, Index
– Index objects represent legacy (v1.0) indexes
– v2.0 “background” indexes handled in Schema
• Are blessed scalar refs : "Inside-out object" pattern
– the scalar value is the item ID (or index name)
– For any object $obj, $$obj (the ID) is exactly what you need
for constructing the API calls
• Are subclasses of Entity
– Entity does the object table handling, JSON-to-object
conversion and HTTP agent calls
– Isolates most of the kludges necessary to handle the few
API inconsistencies that exist(ed)
14
Batch Calls
• Certain situations (database loading, e.g.)
make sense to batch : do many things in one
API call rather than many single calls
• REST API provides this functionality
• How to make it "natural" in the context of
working with objects?
– Use Perl prototyping sugar to create a "batch
block"
15
Example:
Rather than call the server for every line, you can mix in
REST::Neo4p::Batch, and then use a batch {} block:
16
Calls within
block are
collected and
deferred
17
You can execute more complex logic within the
batch block, and keep the objects beyond it:
18
But miracles are not yet implemented:
Object here doesn't really exist yet…
How does that work?
• Agent module isolates all bona fide calls
– very few kludges to core object modules req'd
• batch() puts the agent into “batch mode” and
executes wrapped code
– agent stores incoming calls as JSON in a queue
• After wrapped code is executed, batch() switches
agent back to normal mode and has it call the
batch endpoint with the queue contents
• Batch processes the response and creates objects
if requested
19
HTTP Agent
20
Agent
• Is transparent
– But can always see it with REST::Neo4p->agent
– Agent module alone meant to be useful and independent
• Elicits and uses the API self-discovery feature on
connect()
• Isolates all HTTP requests and responses
• Captures and distinguishes API and HTTP errors
– emits REST::Neo4p::Exceptions objects
• [Instance] Is a subclass of a "real" user agent:
– LWP::UserAgent
– Mojo::UserAgent, or
– HTTP::Thin
21
Working within API Self-Description
22
• Get the list of actions with
– $agent->available_actions
• And AUTOLOAD will provide (see pod for args):
– $agent->get_<action>()
– $agent->put_<action>()
– $agent->post_<action>()
– $agent->delete_<action>()
• Other accessors, e.g. node(), return the
appropriate URL for your server
Schemas - Use Case
You start out with a set of well categorized things, that
have some well defined relationships.
Each thing will be represented as a node, that's fine. But,
You want to guarantee (to your client, for example) that
1. You can classify every node you add or read
unambiguously into a well-defined group
(you know everything that’s in there);
2. You never relate two nodes belonging to particular
groups in a way that doesn't make sense according to
your well-defined relationships
(you can find everything that’s in there).
23
Schema Helps
• REST::Neo4p::Schema – Access the (limited)
schema functionality of Neo4j server
– Create indexes
– Maintain uniqueness of nodes within Label classes
• REST::Neo4p::Constrain - An add-in for
constraining (or validating)
– property values
– connections (relationships) based on node properties
– relationship types
according to flexible specifications
24
App-level Constraints
25
26
27
Will throw at
Record 5
Constrain/Constraint
• Multiple modes:
– Automatic (throws exception if constraint
violated)
– Manual (validation function returns false if
constraint violated)
– Suspended (lift constraint processing when
desired)
• Freeze/Thaw (in JSON) constraint
specifications for reuse
28
Cypher Queries
• REST::Neo4p::Query takes a familiar, DBI-like
approach
– Prepare, execute, fetch
– "rows" returned are arrays containing scalars,
Node objects, and/or Relationship objects
• Simple Perl data structures can be requested instead if
desired
– If a query returns a path, a Path object (a simple
container) is returned
29
30
Cypher Queries
• Prepare and execute with parameter substitutions
31
Do This!
Not This!
Cypher Queries
• Transactions are supported when you have
v2.0.1 server or greater
– started with REST::Neo4p->begin_work()
– committed with REST::Neo4p->commit()
– canceled with REST::Neo4p->rollback()
(here, the class looks like the database handle in
DBI, in fact…)
32
DBI – DBD::Neo4p
• Yes, you can really do this:
33
DBI – DBD::Neo4p
34
• Row returns: choice of full objects or simple
Perl structures
Future Directions/Contribution Ideas
• Test on v2.2 server and fix any issues
• Make Neo4p closer to an ORM (require explicit
push/pull from backend server)
• Sunset v1.0 support
– Completely touch-free testing within transactions
– Integrate node labels better
• Make batch response parsing more efficient
– e.g., don't stream if response is not huge
• Add traversal functionality
• Beautify and deodorize
35
Thanks!
36
https://github.com/majensen/rest-neo4p.git

More Related Content

What's hot

API Development with Laravel
API Development with LaravelAPI Development with Laravel
API Development with Laravel
Michael Peacock
 
FlutterでGraphQLを扱う
FlutterでGraphQLを扱うFlutterでGraphQLを扱う
FlutterでGraphQLを扱う
IgaHironobu
 
#Pharo Days 2016 Data Formats and Protocols
#Pharo Days 2016 Data Formats and Protocols#Pharo Days 2016 Data Formats and Protocols
#Pharo Days 2016 Data Formats and Protocols
Philippe Back
 
PuppetConf 2017: Custom Types & Providers: Modeling Modern REST Interfaces an...
PuppetConf 2017: Custom Types & Providers: Modeling Modern REST Interfaces an...PuppetConf 2017: Custom Types & Providers: Modeling Modern REST Interfaces an...
PuppetConf 2017: Custom Types & Providers: Modeling Modern REST Interfaces an...
Puppet
 
Laravel and Django and Rails, Oh My!
Laravel and Django and Rails, Oh My!Laravel and Django and Rails, Oh My!
Laravel and Django and Rails, Oh My!
Chris Roberts
 
Oak, the architecture of Apache Jackrabbit 3
Oak, the architecture of Apache Jackrabbit 3Oak, the architecture of Apache Jackrabbit 3
Oak, the architecture of Apache Jackrabbit 3
Jukka Zitting
 
Ractor's speed is not light-speed
Ractor's speed is not light-speedRactor's speed is not light-speed
Ractor's speed is not light-speed
SATOSHI TAGOMORI
 
Java JSON Parser Comparison
Java JSON Parser ComparisonJava JSON Parser Comparison
Java JSON Parser Comparison
Allan Huang
 
Laravel 5 Annotations: RESTful API routing
Laravel 5 Annotations: RESTful API routingLaravel 5 Annotations: RESTful API routing
Laravel 5 Annotations: RESTful API routing
Christopher Pecoraro
 
RxJava from the trenches
RxJava from the trenchesRxJava from the trenches
RxJava from the trenches
Peter Hendriks
 
Data collection in AWS at Schibsted
Data collection in AWS at SchibstedData collection in AWS at Schibsted
Data collection in AWS at Schibsted
Lars Marius Garshol
 
Stream processing from single node to a cluster
Stream processing from single node to a clusterStream processing from single node to a cluster
Stream processing from single node to a cluster
Gal Marder
 
Loading 350M documents into a large Solr cluster: Presented by Dion Olsthoorn...
Loading 350M documents into a large Solr cluster: Presented by Dion Olsthoorn...Loading 350M documents into a large Solr cluster: Presented by Dion Olsthoorn...
Loading 350M documents into a large Solr cluster: Presented by Dion Olsthoorn...
Lucidworks
 
Building Distributed Systems in Scala
Building Distributed Systems in ScalaBuilding Distributed Systems in Scala
Building Distributed Systems in Scala
Alex Payne
 
Ruby on the JVM
Ruby on the JVMRuby on the JVM
Ruby on the JVM
Kresten Krab Thorup
 
Open Ldap Integration and Configuration with Lifray 6.2
Open Ldap Integration and Configuration with Lifray 6.2Open Ldap Integration and Configuration with Lifray 6.2
Open Ldap Integration and Configuration with Lifray 6.2
Vinaykumar Hebballi
 
A introduction to Laravel framework
A introduction to Laravel frameworkA introduction to Laravel framework
A introduction to Laravel framework
Phu Luong Trong
 
libAttachSQL, The Next-Generation C Connector For MySQL
libAttachSQL, The Next-Generation C Connector For MySQLlibAttachSQL, The Next-Generation C Connector For MySQL
libAttachSQL, The Next-Generation C Connector For MySQL
Andrew Hutchings
 
I18nize Scala programs à la gettext
I18nize Scala programs à la gettextI18nize Scala programs à la gettext
I18nize Scala programs à la gettext
Ngoc Dao
 
Solr Troubleshooting - Treemap Approach: Presented by Alexandre Rafolovitch, ...
Solr Troubleshooting - Treemap Approach: Presented by Alexandre Rafolovitch, ...Solr Troubleshooting - Treemap Approach: Presented by Alexandre Rafolovitch, ...
Solr Troubleshooting - Treemap Approach: Presented by Alexandre Rafolovitch, ...
Lucidworks
 

What's hot (20)

API Development with Laravel
API Development with LaravelAPI Development with Laravel
API Development with Laravel
 
FlutterでGraphQLを扱う
FlutterでGraphQLを扱うFlutterでGraphQLを扱う
FlutterでGraphQLを扱う
 
#Pharo Days 2016 Data Formats and Protocols
#Pharo Days 2016 Data Formats and Protocols#Pharo Days 2016 Data Formats and Protocols
#Pharo Days 2016 Data Formats and Protocols
 
PuppetConf 2017: Custom Types & Providers: Modeling Modern REST Interfaces an...
PuppetConf 2017: Custom Types & Providers: Modeling Modern REST Interfaces an...PuppetConf 2017: Custom Types & Providers: Modeling Modern REST Interfaces an...
PuppetConf 2017: Custom Types & Providers: Modeling Modern REST Interfaces an...
 
Laravel and Django and Rails, Oh My!
Laravel and Django and Rails, Oh My!Laravel and Django and Rails, Oh My!
Laravel and Django and Rails, Oh My!
 
Oak, the architecture of Apache Jackrabbit 3
Oak, the architecture of Apache Jackrabbit 3Oak, the architecture of Apache Jackrabbit 3
Oak, the architecture of Apache Jackrabbit 3
 
Ractor's speed is not light-speed
Ractor's speed is not light-speedRactor's speed is not light-speed
Ractor's speed is not light-speed
 
Java JSON Parser Comparison
Java JSON Parser ComparisonJava JSON Parser Comparison
Java JSON Parser Comparison
 
Laravel 5 Annotations: RESTful API routing
Laravel 5 Annotations: RESTful API routingLaravel 5 Annotations: RESTful API routing
Laravel 5 Annotations: RESTful API routing
 
RxJava from the trenches
RxJava from the trenchesRxJava from the trenches
RxJava from the trenches
 
Data collection in AWS at Schibsted
Data collection in AWS at SchibstedData collection in AWS at Schibsted
Data collection in AWS at Schibsted
 
Stream processing from single node to a cluster
Stream processing from single node to a clusterStream processing from single node to a cluster
Stream processing from single node to a cluster
 
Loading 350M documents into a large Solr cluster: Presented by Dion Olsthoorn...
Loading 350M documents into a large Solr cluster: Presented by Dion Olsthoorn...Loading 350M documents into a large Solr cluster: Presented by Dion Olsthoorn...
Loading 350M documents into a large Solr cluster: Presented by Dion Olsthoorn...
 
Building Distributed Systems in Scala
Building Distributed Systems in ScalaBuilding Distributed Systems in Scala
Building Distributed Systems in Scala
 
Ruby on the JVM
Ruby on the JVMRuby on the JVM
Ruby on the JVM
 
Open Ldap Integration and Configuration with Lifray 6.2
Open Ldap Integration and Configuration with Lifray 6.2Open Ldap Integration and Configuration with Lifray 6.2
Open Ldap Integration and Configuration with Lifray 6.2
 
A introduction to Laravel framework
A introduction to Laravel frameworkA introduction to Laravel framework
A introduction to Laravel framework
 
libAttachSQL, The Next-Generation C Connector For MySQL
libAttachSQL, The Next-Generation C Connector For MySQLlibAttachSQL, The Next-Generation C Connector For MySQL
libAttachSQL, The Next-Generation C Connector For MySQL
 
I18nize Scala programs à la gettext
I18nize Scala programs à la gettextI18nize Scala programs à la gettext
I18nize Scala programs à la gettext
 
Solr Troubleshooting - Treemap Approach: Presented by Alexandre Rafolovitch, ...
Solr Troubleshooting - Treemap Approach: Presented by Alexandre Rafolovitch, ...Solr Troubleshooting - Treemap Approach: Presented by Alexandre Rafolovitch, ...
Solr Troubleshooting - Treemap Approach: Presented by Alexandre Rafolovitch, ...
 

Viewers also liked

Language show 2010 languages work presentation
Language show 2010 languages work presentationLanguage show 2010 languages work presentation
Language show 2010 languages work presentation
RoisinMcGinley
 
NDCC Presentation Final
NDCC Presentation FinalNDCC Presentation Final
NDCC Presentation Final
laureen920
 
OSS Community management
OSS Community managementOSS Community management
A recipe for happiness 2
A recipe for happiness 2A recipe for happiness 2
A recipe for happiness 2
mariah4everpeace
 
Choisi Promo
Choisi PromoChoisi Promo
Choisi Promo
Menzelle Jacobs
 
#SAIFC késako - Semantic Analysis for Flow Computing
 #SAIFC késako - Semantic Analysis for Flow Computing #SAIFC késako - Semantic Analysis for Flow Computing
#SAIFC késako - Semantic Analysis for Flow Computing
fOSSa - Free Open Source Software Academia Conference
 
Discipline
DisciplineDiscipline
Discipline
9950012133
 
Sports marketing: where does the future lie - in the content or the conversa...
Sports marketing:  where does the future lie - in the content or the conversa...Sports marketing:  where does the future lie - in the content or the conversa...
Sports marketing: where does the future lie - in the content or the conversa...
cainster
 
NYC Benchmarking Seminar: Aggregated Consumption Data
NYC Benchmarking Seminar: Aggregated Consumption DataNYC Benchmarking Seminar: Aggregated Consumption Data
NYC Benchmarking Seminar: Aggregated Consumption Data
REBNY
 
Raicesdeecuaciones
RaicesdeecuacionesRaicesdeecuaciones
Raicesdeecuaciones
uis
 
Celik alquran Part 1
Celik alquran Part 1Celik alquran Part 1
Celik alquran Part 1mimi
 
Hy solution사례(5)저축보험가입고객
Hy solution사례(5)저축보험가입고객Hy solution사례(5)저축보험가입고객
Hy solution사례(5)저축보험가입고객
valuasset
 
Eco System over code!
Eco System over code!Eco System over code!
EPM Live PortfolioEngine
EPM Live PortfolioEngineEPM Live PortfolioEngine
EPM Live PortfolioEngine
EPM Live
 
Nyc bid conference
Nyc bid conferenceNyc bid conference
Nyc bid conference
REBNY
 
Nathan Ogle AIA LEED AP Portfolio Highlights 2012
Nathan Ogle AIA LEED AP Portfolio Highlights 2012Nathan Ogle AIA LEED AP Portfolio Highlights 2012
Nathan Ogle AIA LEED AP Portfolio Highlights 2012
Nathan Ogle
 
REBNY NYC Benchmarking Seminar: NYSERDA ncentive Programs for Multi‐family Bu...
REBNY NYC Benchmarking Seminar: NYSERDA ncentive Programs for Multi‐family Bu...REBNY NYC Benchmarking Seminar: NYSERDA ncentive Programs for Multi‐family Bu...
REBNY NYC Benchmarking Seminar: NYSERDA ncentive Programs for Multi‐family Bu...
REBNY
 
Shawn.trela ppt
Shawn.trela pptShawn.trela ppt
Shawn.trela ppt
trela
 
Autacoids1
Autacoids1Autacoids1
Autacoids1
shahhmurad
 

Viewers also liked (20)

Language show 2010 languages work presentation
Language show 2010 languages work presentationLanguage show 2010 languages work presentation
Language show 2010 languages work presentation
 
NDCC Presentation Final
NDCC Presentation FinalNDCC Presentation Final
NDCC Presentation Final
 
OSS Community management
OSS Community managementOSS Community management
OSS Community management
 
A recipe for happiness 2
A recipe for happiness 2A recipe for happiness 2
A recipe for happiness 2
 
Choisi Promo
Choisi PromoChoisi Promo
Choisi Promo
 
#SAIFC késako - Semantic Analysis for Flow Computing
 #SAIFC késako - Semantic Analysis for Flow Computing #SAIFC késako - Semantic Analysis for Flow Computing
#SAIFC késako - Semantic Analysis for Flow Computing
 
Discipline
DisciplineDiscipline
Discipline
 
Sports marketing: where does the future lie - in the content or the conversa...
Sports marketing:  where does the future lie - in the content or the conversa...Sports marketing:  where does the future lie - in the content or the conversa...
Sports marketing: where does the future lie - in the content or the conversa...
 
NYC Benchmarking Seminar: Aggregated Consumption Data
NYC Benchmarking Seminar: Aggregated Consumption DataNYC Benchmarking Seminar: Aggregated Consumption Data
NYC Benchmarking Seminar: Aggregated Consumption Data
 
Raicesdeecuaciones
RaicesdeecuacionesRaicesdeecuaciones
Raicesdeecuaciones
 
Celik alquran Part 1
Celik alquran Part 1Celik alquran Part 1
Celik alquran Part 1
 
Hy solution사례(5)저축보험가입고객
Hy solution사례(5)저축보험가입고객Hy solution사례(5)저축보험가입고객
Hy solution사례(5)저축보험가입고객
 
Eco System over code!
Eco System over code!Eco System over code!
Eco System over code!
 
EPM Live PortfolioEngine
EPM Live PortfolioEngineEPM Live PortfolioEngine
EPM Live PortfolioEngine
 
Nyc bid conference
Nyc bid conferenceNyc bid conference
Nyc bid conference
 
Nathan Ogle AIA LEED AP Portfolio Highlights 2012
Nathan Ogle AIA LEED AP Portfolio Highlights 2012Nathan Ogle AIA LEED AP Portfolio Highlights 2012
Nathan Ogle AIA LEED AP Portfolio Highlights 2012
 
REBNY NYC Benchmarking Seminar: NYSERDA ncentive Programs for Multi‐family Bu...
REBNY NYC Benchmarking Seminar: NYSERDA ncentive Programs for Multi‐family Bu...REBNY NYC Benchmarking Seminar: NYSERDA ncentive Programs for Multi‐family Bu...
REBNY NYC Benchmarking Seminar: NYSERDA ncentive Programs for Multi‐family Bu...
 
Shawn.trela ppt
Shawn.trela pptShawn.trela ppt
Shawn.trela ppt
 
Autacoids1
Autacoids1Autacoids1
Autacoids1
 
Jurnal Indonesia Review - Agustus 2010
Jurnal Indonesia Review - Agustus 2010Jurnal Indonesia Review - Agustus 2010
Jurnal Indonesia Review - Agustus 2010
 

Similar to Neo4p dcbpw-2015

Microservices for Systematic Profiling and Monitoring of the Refactoring
Microservices for Systematic Profiling and Monitoring of the RefactoringMicroservices for Systematic Profiling and Monitoring of the Refactoring
Microservices for Systematic Profiling and Monitoring of the Refactoring
Alexander Mazurov
 
ElasticSearch AJUG 2013
ElasticSearch AJUG 2013ElasticSearch AJUG 2013
ElasticSearch AJUG 2013
Roy Russo
 
Arabidopsis Information Portal, Developer Workshop 2014, Introduction
Arabidopsis Information Portal, Developer Workshop 2014, IntroductionArabidopsis Information Portal, Developer Workshop 2014, Introduction
Arabidopsis Information Portal, Developer Workshop 2014, Introduction
JasonRafeMiller
 
Ruby on-rails-101-presentation-slides-for-a-five-day-introductory-course-1194...
Ruby on-rails-101-presentation-slides-for-a-five-day-introductory-course-1194...Ruby on-rails-101-presentation-slides-for-a-five-day-introductory-course-1194...
Ruby on-rails-101-presentation-slides-for-a-five-day-introductory-course-1194...
Nilesh Panchal
 
Java Web services
Java Web servicesJava Web services
Java Web services
Sujit Kumar
 
Kubernetes intro public - kubernetes meetup 4-21-2015
Kubernetes intro   public - kubernetes meetup 4-21-2015Kubernetes intro   public - kubernetes meetup 4-21-2015
Kubernetes intro public - kubernetes meetup 4-21-2015
Rohit Jnagal
 
Kubernetes intro public - kubernetes user group 4-21-2015
Kubernetes intro   public - kubernetes user group 4-21-2015Kubernetes intro   public - kubernetes user group 4-21-2015
Kubernetes intro public - kubernetes user group 4-21-2015
reallavalamp
 
MongoDB: a gentle, friendly overview
MongoDB: a gentle, friendly overviewMongoDB: a gentle, friendly overview
MongoDB: a gentle, friendly overview
Antonio Pintus
 
"Data Provenance: Principles and Why it matters for BioMedical Applications"
"Data Provenance: Principles and Why it matters for BioMedical Applications""Data Provenance: Principles and Why it matters for BioMedical Applications"
"Data Provenance: Principles and Why it matters for BioMedical Applications"
Pinar Alper
 
NoSQL, Apache SOLR and Apache Hadoop
NoSQL, Apache SOLR and Apache HadoopNoSQL, Apache SOLR and Apache Hadoop
NoSQL, Apache SOLR and Apache Hadoop
Dmitry Kan
 
REST APIs
REST APIsREST APIs
Dev nexus 2017
Dev nexus 2017Dev nexus 2017
Dev nexus 2017
Roy Russo
 
Impala Architecture presentation
Impala Architecture presentationImpala Architecture presentation
Impala Architecture presentation
hadooparchbook
 
Lessons learned while building Omroep.nl
Lessons learned while building Omroep.nlLessons learned while building Omroep.nl
Lessons learned while building Omroep.nl
bartzon
 
Apache Kylin: OLAP Engine on Hadoop - Tech Deep Dive
Apache Kylin: OLAP Engine on Hadoop - Tech Deep DiveApache Kylin: OLAP Engine on Hadoop - Tech Deep Dive
Apache Kylin: OLAP Engine on Hadoop - Tech Deep Dive
Xu Jiang
 
JCR - Java Content Repositories
JCR - Java Content RepositoriesJCR - Java Content Repositories
JCR - Java Content Repositories
Carsten Ziegeler
 
RESTful Web Service using Swagger
RESTful Web Service using SwaggerRESTful Web Service using Swagger
RESTful Web Service using Swagger
Hong-Jhih Lin
 
Lessons learned while building Omroep.nl
Lessons learned while building Omroep.nlLessons learned while building Omroep.nl
Lessons learned while building Omroep.nl
tieleman
 
Restful风格ž„web服务架构
Restful风格ž„web服务架构Restful风格ž„web服务架构
Restful风格ž„web服务架构
Benjamin Tan
 
Introduction to Laravel Framework (5.2)
Introduction to Laravel Framework (5.2)Introduction to Laravel Framework (5.2)
Introduction to Laravel Framework (5.2)
Viral Solani
 

Similar to Neo4p dcbpw-2015 (20)

Microservices for Systematic Profiling and Monitoring of the Refactoring
Microservices for Systematic Profiling and Monitoring of the RefactoringMicroservices for Systematic Profiling and Monitoring of the Refactoring
Microservices for Systematic Profiling and Monitoring of the Refactoring
 
ElasticSearch AJUG 2013
ElasticSearch AJUG 2013ElasticSearch AJUG 2013
ElasticSearch AJUG 2013
 
Arabidopsis Information Portal, Developer Workshop 2014, Introduction
Arabidopsis Information Portal, Developer Workshop 2014, IntroductionArabidopsis Information Portal, Developer Workshop 2014, Introduction
Arabidopsis Information Portal, Developer Workshop 2014, Introduction
 
Ruby on-rails-101-presentation-slides-for-a-five-day-introductory-course-1194...
Ruby on-rails-101-presentation-slides-for-a-five-day-introductory-course-1194...Ruby on-rails-101-presentation-slides-for-a-five-day-introductory-course-1194...
Ruby on-rails-101-presentation-slides-for-a-five-day-introductory-course-1194...
 
Java Web services
Java Web servicesJava Web services
Java Web services
 
Kubernetes intro public - kubernetes meetup 4-21-2015
Kubernetes intro   public - kubernetes meetup 4-21-2015Kubernetes intro   public - kubernetes meetup 4-21-2015
Kubernetes intro public - kubernetes meetup 4-21-2015
 
Kubernetes intro public - kubernetes user group 4-21-2015
Kubernetes intro   public - kubernetes user group 4-21-2015Kubernetes intro   public - kubernetes user group 4-21-2015
Kubernetes intro public - kubernetes user group 4-21-2015
 
MongoDB: a gentle, friendly overview
MongoDB: a gentle, friendly overviewMongoDB: a gentle, friendly overview
MongoDB: a gentle, friendly overview
 
"Data Provenance: Principles and Why it matters for BioMedical Applications"
"Data Provenance: Principles and Why it matters for BioMedical Applications""Data Provenance: Principles and Why it matters for BioMedical Applications"
"Data Provenance: Principles and Why it matters for BioMedical Applications"
 
NoSQL, Apache SOLR and Apache Hadoop
NoSQL, Apache SOLR and Apache HadoopNoSQL, Apache SOLR and Apache Hadoop
NoSQL, Apache SOLR and Apache Hadoop
 
REST APIs
REST APIsREST APIs
REST APIs
 
Dev nexus 2017
Dev nexus 2017Dev nexus 2017
Dev nexus 2017
 
Impala Architecture presentation
Impala Architecture presentationImpala Architecture presentation
Impala Architecture presentation
 
Lessons learned while building Omroep.nl
Lessons learned while building Omroep.nlLessons learned while building Omroep.nl
Lessons learned while building Omroep.nl
 
Apache Kylin: OLAP Engine on Hadoop - Tech Deep Dive
Apache Kylin: OLAP Engine on Hadoop - Tech Deep DiveApache Kylin: OLAP Engine on Hadoop - Tech Deep Dive
Apache Kylin: OLAP Engine on Hadoop - Tech Deep Dive
 
JCR - Java Content Repositories
JCR - Java Content RepositoriesJCR - Java Content Repositories
JCR - Java Content Repositories
 
RESTful Web Service using Swagger
RESTful Web Service using SwaggerRESTful Web Service using Swagger
RESTful Web Service using Swagger
 
Lessons learned while building Omroep.nl
Lessons learned while building Omroep.nlLessons learned while building Omroep.nl
Lessons learned while building Omroep.nl
 
Restful风格ž„web服务架构
Restful风格ž„web服务架构Restful风格ž„web服务架构
Restful风格ž„web服务架构
 
Introduction to Laravel Framework (5.2)
Introduction to Laravel Framework (5.2)Introduction to Laravel Framework (5.2)
Introduction to Laravel Framework (5.2)
 

Recently uploaded

GreenCode-A-VSCode-Plugin--Dario-Jurisic
GreenCode-A-VSCode-Plugin--Dario-JurisicGreenCode-A-VSCode-Plugin--Dario-Jurisic
GreenCode-A-VSCode-Plugin--Dario-Jurisic
Green Software Development
 
Mobile App Development Company In Noida | Drona Infotech
Mobile App Development Company In Noida | Drona InfotechMobile App Development Company In Noida | Drona Infotech
Mobile App Development Company In Noida | Drona Infotech
Drona Infotech
 
How Can Hiring A Mobile App Development Company Help Your Business Grow?
How Can Hiring A Mobile App Development Company Help Your Business Grow?How Can Hiring A Mobile App Development Company Help Your Business Grow?
How Can Hiring A Mobile App Development Company Help Your Business Grow?
ToXSL Technologies
 
在线购买加拿大英属哥伦比亚大学毕业证本科学位证书原版一模一样
在线购买加拿大英属哥伦比亚大学毕业证本科学位证书原版一模一样在线购买加拿大英属哥伦比亚大学毕业证本科学位证书原版一模一样
在线购买加拿大英属哥伦比亚大学毕业证本科学位证书原版一模一样
mz5nrf0n
 
Webinar On-Demand: Using Flutter for Embedded
Webinar On-Demand: Using Flutter for EmbeddedWebinar On-Demand: Using Flutter for Embedded
Webinar On-Demand: Using Flutter for Embedded
ICS
 
How to write a program in any programming language
How to write a program in any programming languageHow to write a program in any programming language
How to write a program in any programming language
Rakesh Kumar R
 
What next after learning python programming basics
What next after learning python programming basicsWhat next after learning python programming basics
What next after learning python programming basics
Rakesh Kumar R
 
Everything You Need to Know About X-Sign: The eSign Functionality of XfilesPr...
Everything You Need to Know About X-Sign: The eSign Functionality of XfilesPr...Everything You Need to Know About X-Sign: The eSign Functionality of XfilesPr...
Everything You Need to Know About X-Sign: The eSign Functionality of XfilesPr...
XfilesPro
 
一比一原版(USF毕业证)旧金山大学毕业证如何办理
一比一原版(USF毕业证)旧金山大学毕业证如何办理一比一原版(USF毕业证)旧金山大学毕业证如何办理
一比一原版(USF毕业证)旧金山大学毕业证如何办理
dakas1
 
E-commerce Development Services- Hornet Dynamics
E-commerce Development Services- Hornet DynamicsE-commerce Development Services- Hornet Dynamics
E-commerce Development Services- Hornet Dynamics
Hornet Dynamics
 
socradar-q1-2024-aviation-industry-report.pdf
socradar-q1-2024-aviation-industry-report.pdfsocradar-q1-2024-aviation-industry-report.pdf
socradar-q1-2024-aviation-industry-report.pdf
SOCRadar
 
Transform Your Communication with Cloud-Based IVR Solutions
Transform Your Communication with Cloud-Based IVR SolutionsTransform Your Communication with Cloud-Based IVR Solutions
Transform Your Communication with Cloud-Based IVR Solutions
TheSMSPoint
 
Odoo ERP Vs. Traditional ERP Systems – A Comparative Analysis
Odoo ERP Vs. Traditional ERP Systems – A Comparative AnalysisOdoo ERP Vs. Traditional ERP Systems – A Comparative Analysis
Odoo ERP Vs. Traditional ERP Systems – A Comparative Analysis
Envertis Software Solutions
 
All you need to know about Spring Boot and GraalVM
All you need to know about Spring Boot and GraalVMAll you need to know about Spring Boot and GraalVM
All you need to know about Spring Boot and GraalVM
Alina Yurenko
 
SMS API Integration in Saudi Arabia| Best SMS API Service
SMS API Integration in Saudi Arabia| Best SMS API ServiceSMS API Integration in Saudi Arabia| Best SMS API Service
SMS API Integration in Saudi Arabia| Best SMS API Service
Yara Milbes
 
Using Xen Hypervisor for Functional Safety
Using Xen Hypervisor for Functional SafetyUsing Xen Hypervisor for Functional Safety
Using Xen Hypervisor for Functional Safety
Ayan Halder
 
E-Invoicing Implementation: A Step-by-Step Guide for Saudi Arabian Companies
E-Invoicing Implementation: A Step-by-Step Guide for Saudi Arabian CompaniesE-Invoicing Implementation: A Step-by-Step Guide for Saudi Arabian Companies
E-Invoicing Implementation: A Step-by-Step Guide for Saudi Arabian Companies
Quickdice ERP
 
Need for Speed: Removing speed bumps from your Symfony projects ⚡️
Need for Speed: Removing speed bumps from your Symfony projects ⚡️Need for Speed: Removing speed bumps from your Symfony projects ⚡️
Need for Speed: Removing speed bumps from your Symfony projects ⚡️
Łukasz Chruściel
 
UI5con 2024 - Keynote: Latest News about UI5 and it’s Ecosystem
UI5con 2024 - Keynote: Latest News about UI5 and it’s EcosystemUI5con 2024 - Keynote: Latest News about UI5 and it’s Ecosystem
UI5con 2024 - Keynote: Latest News about UI5 and it’s Ecosystem
Peter Muessig
 
Oracle Database 19c New Features for DBAs and Developers.pptx
Oracle Database 19c New Features for DBAs and Developers.pptxOracle Database 19c New Features for DBAs and Developers.pptx
Oracle Database 19c New Features for DBAs and Developers.pptx
Remote DBA Services
 

Recently uploaded (20)

GreenCode-A-VSCode-Plugin--Dario-Jurisic
GreenCode-A-VSCode-Plugin--Dario-JurisicGreenCode-A-VSCode-Plugin--Dario-Jurisic
GreenCode-A-VSCode-Plugin--Dario-Jurisic
 
Mobile App Development Company In Noida | Drona Infotech
Mobile App Development Company In Noida | Drona InfotechMobile App Development Company In Noida | Drona Infotech
Mobile App Development Company In Noida | Drona Infotech
 
How Can Hiring A Mobile App Development Company Help Your Business Grow?
How Can Hiring A Mobile App Development Company Help Your Business Grow?How Can Hiring A Mobile App Development Company Help Your Business Grow?
How Can Hiring A Mobile App Development Company Help Your Business Grow?
 
在线购买加拿大英属哥伦比亚大学毕业证本科学位证书原版一模一样
在线购买加拿大英属哥伦比亚大学毕业证本科学位证书原版一模一样在线购买加拿大英属哥伦比亚大学毕业证本科学位证书原版一模一样
在线购买加拿大英属哥伦比亚大学毕业证本科学位证书原版一模一样
 
Webinar On-Demand: Using Flutter for Embedded
Webinar On-Demand: Using Flutter for EmbeddedWebinar On-Demand: Using Flutter for Embedded
Webinar On-Demand: Using Flutter for Embedded
 
How to write a program in any programming language
How to write a program in any programming languageHow to write a program in any programming language
How to write a program in any programming language
 
What next after learning python programming basics
What next after learning python programming basicsWhat next after learning python programming basics
What next after learning python programming basics
 
Everything You Need to Know About X-Sign: The eSign Functionality of XfilesPr...
Everything You Need to Know About X-Sign: The eSign Functionality of XfilesPr...Everything You Need to Know About X-Sign: The eSign Functionality of XfilesPr...
Everything You Need to Know About X-Sign: The eSign Functionality of XfilesPr...
 
一比一原版(USF毕业证)旧金山大学毕业证如何办理
一比一原版(USF毕业证)旧金山大学毕业证如何办理一比一原版(USF毕业证)旧金山大学毕业证如何办理
一比一原版(USF毕业证)旧金山大学毕业证如何办理
 
E-commerce Development Services- Hornet Dynamics
E-commerce Development Services- Hornet DynamicsE-commerce Development Services- Hornet Dynamics
E-commerce Development Services- Hornet Dynamics
 
socradar-q1-2024-aviation-industry-report.pdf
socradar-q1-2024-aviation-industry-report.pdfsocradar-q1-2024-aviation-industry-report.pdf
socradar-q1-2024-aviation-industry-report.pdf
 
Transform Your Communication with Cloud-Based IVR Solutions
Transform Your Communication with Cloud-Based IVR SolutionsTransform Your Communication with Cloud-Based IVR Solutions
Transform Your Communication with Cloud-Based IVR Solutions
 
Odoo ERP Vs. Traditional ERP Systems – A Comparative Analysis
Odoo ERP Vs. Traditional ERP Systems – A Comparative AnalysisOdoo ERP Vs. Traditional ERP Systems – A Comparative Analysis
Odoo ERP Vs. Traditional ERP Systems – A Comparative Analysis
 
All you need to know about Spring Boot and GraalVM
All you need to know about Spring Boot and GraalVMAll you need to know about Spring Boot and GraalVM
All you need to know about Spring Boot and GraalVM
 
SMS API Integration in Saudi Arabia| Best SMS API Service
SMS API Integration in Saudi Arabia| Best SMS API ServiceSMS API Integration in Saudi Arabia| Best SMS API Service
SMS API Integration in Saudi Arabia| Best SMS API Service
 
Using Xen Hypervisor for Functional Safety
Using Xen Hypervisor for Functional SafetyUsing Xen Hypervisor for Functional Safety
Using Xen Hypervisor for Functional Safety
 
E-Invoicing Implementation: A Step-by-Step Guide for Saudi Arabian Companies
E-Invoicing Implementation: A Step-by-Step Guide for Saudi Arabian CompaniesE-Invoicing Implementation: A Step-by-Step Guide for Saudi Arabian Companies
E-Invoicing Implementation: A Step-by-Step Guide for Saudi Arabian Companies
 
Need for Speed: Removing speed bumps from your Symfony projects ⚡️
Need for Speed: Removing speed bumps from your Symfony projects ⚡️Need for Speed: Removing speed bumps from your Symfony projects ⚡️
Need for Speed: Removing speed bumps from your Symfony projects ⚡️
 
UI5con 2024 - Keynote: Latest News about UI5 and it’s Ecosystem
UI5con 2024 - Keynote: Latest News about UI5 and it’s EcosystemUI5con 2024 - Keynote: Latest News about UI5 and it’s Ecosystem
UI5con 2024 - Keynote: Latest News about UI5 and it’s Ecosystem
 
Oracle Database 19c New Features for DBAs and Developers.pptx
Oracle Database 19c New Features for DBAs and Developers.pptxOracle Database 19c New Features for DBAs and Developers.pptx
Oracle Database 19c New Features for DBAs and Developers.pptx
 

Neo4p dcbpw-2015

  • 2. • Perler since 2000 • CPAN contributor (MAJENSEN) since 2009 • BioPerl Core Developer • Director, Genomic Data Programs, Leidos Biomedical Research Inc (FNLCR) • @thinkinator, LinkedIn 2
  • 3. Not my sponsor, but could be yours! • http://www.perlfoundation.org/how_to_write_a_proposal 3
  • 4. Motivation • Cancer Genomics: Biospecimen, Clinical, Analysis Data – complex – growing – evolving technologies – evolving policies – need for precise accounting • Graph models are well-suited to this world 4
  • 7. 7 select bar.name from bar, bar_baz, baz, baz_goob, goob, goob_squirrel, squirrel, squirrel_spam, spam, spam_eggs, eggs, eggs_foo, foo where bar.id = bar_baz.bar_id and bar_baz.baz_id = baz.id and baz.id = baz_goob.baz_id and baz_goob.goob_id = goob.id and goob.id = goob_squirrel.goob_id and goob_squirrel.id = squirrel.id and squirrel.id = squirrel_spam.squirrel_id and squirrel_spam.spam_id = spam.id and spam.id = spam_eggs.spam_id and spam_eggs.eggs_id = eggs.id and eggs_foo.eggs_id = eggs.id and eggs_foo.foo_id = foo.id and foo.name = 'zloty'; match (f:foo)-[*5..8]-(b:bar) where f.name = 'zloty' return b.name
  • 8. Neo4j • “Native” graph DB engine (currently in v2.2) – Written in Java, but – Very complete REST API – Custom query language: Cypher – Free community edition – Lots of community support, including many “language drivers” • Not the only one out there, but probably the most widely used (certainly the best marketed) 8
  • 11. Neo4p 11 Create Node Label Node Create Unique Node Add a Prop Link Nodes Load/Use Index
  • 13. Design Goals • "OGM" – Perl 5 objects backed by the graph • User should never have to deal with a REST endpoint* *Unless she wants to. • User should never/only have to deal with Cypher queries† †Unless he wants/doesn’t want to. • Robust enough for production code – System should approach complete coverage of the REST service – System should be robust to REST API changes and server backward-compatible (or at least version-aware) • Take advantage of the self-describing features of the API 13
  • 14. REST::Neo4p core objects • Are Node, Relationship, Index – Index objects represent legacy (v1.0) indexes – v2.0 “background” indexes handled in Schema • Are blessed scalar refs : "Inside-out object" pattern – the scalar value is the item ID (or index name) – For any object $obj, $$obj (the ID) is exactly what you need for constructing the API calls • Are subclasses of Entity – Entity does the object table handling, JSON-to-object conversion and HTTP agent calls – Isolates most of the kludges necessary to handle the few API inconsistencies that exist(ed) 14
  • 15. Batch Calls • Certain situations (database loading, e.g.) make sense to batch : do many things in one API call rather than many single calls • REST API provides this functionality • How to make it "natural" in the context of working with objects? – Use Perl prototyping sugar to create a "batch block" 15
  • 16. Example: Rather than call the server for every line, you can mix in REST::Neo4p::Batch, and then use a batch {} block: 16 Calls within block are collected and deferred
  • 17. 17 You can execute more complex logic within the batch block, and keep the objects beyond it:
  • 18. 18 But miracles are not yet implemented: Object here doesn't really exist yet…
  • 19. How does that work? • Agent module isolates all bona fide calls – very few kludges to core object modules req'd • batch() puts the agent into “batch mode” and executes wrapped code – agent stores incoming calls as JSON in a queue • After wrapped code is executed, batch() switches agent back to normal mode and has it call the batch endpoint with the queue contents • Batch processes the response and creates objects if requested 19
  • 21. Agent • Is transparent – But can always see it with REST::Neo4p->agent – Agent module alone meant to be useful and independent • Elicits and uses the API self-discovery feature on connect() • Isolates all HTTP requests and responses • Captures and distinguishes API and HTTP errors – emits REST::Neo4p::Exceptions objects • [Instance] Is a subclass of a "real" user agent: – LWP::UserAgent – Mojo::UserAgent, or – HTTP::Thin 21
  • 22. Working within API Self-Description 22 • Get the list of actions with – $agent->available_actions • And AUTOLOAD will provide (see pod for args): – $agent->get_<action>() – $agent->put_<action>() – $agent->post_<action>() – $agent->delete_<action>() • Other accessors, e.g. node(), return the appropriate URL for your server
  • 23. Schemas - Use Case You start out with a set of well categorized things, that have some well defined relationships. Each thing will be represented as a node, that's fine. But, You want to guarantee (to your client, for example) that 1. You can classify every node you add or read unambiguously into a well-defined group (you know everything that’s in there); 2. You never relate two nodes belonging to particular groups in a way that doesn't make sense according to your well-defined relationships (you can find everything that’s in there). 23
  • 24. Schema Helps • REST::Neo4p::Schema – Access the (limited) schema functionality of Neo4j server – Create indexes – Maintain uniqueness of nodes within Label classes • REST::Neo4p::Constrain - An add-in for constraining (or validating) – property values – connections (relationships) based on node properties – relationship types according to flexible specifications 24
  • 26. 26
  • 28. Constrain/Constraint • Multiple modes: – Automatic (throws exception if constraint violated) – Manual (validation function returns false if constraint violated) – Suspended (lift constraint processing when desired) • Freeze/Thaw (in JSON) constraint specifications for reuse 28
  • 29. Cypher Queries • REST::Neo4p::Query takes a familiar, DBI-like approach – Prepare, execute, fetch – "rows" returned are arrays containing scalars, Node objects, and/or Relationship objects • Simple Perl data structures can be requested instead if desired – If a query returns a path, a Path object (a simple container) is returned 29
  • 30. 30
  • 31. Cypher Queries • Prepare and execute with parameter substitutions 31 Do This! Not This!
  • 32. Cypher Queries • Transactions are supported when you have v2.0.1 server or greater – started with REST::Neo4p->begin_work() – committed with REST::Neo4p->commit() – canceled with REST::Neo4p->rollback() (here, the class looks like the database handle in DBI, in fact…) 32
  • 33. DBI – DBD::Neo4p • Yes, you can really do this: 33
  • 34. DBI – DBD::Neo4p 34 • Row returns: choice of full objects or simple Perl structures
  • 35. Future Directions/Contribution Ideas • Test on v2.2 server and fix any issues • Make Neo4p closer to an ORM (require explicit push/pull from backend server) • Sunset v1.0 support – Completely touch-free testing within transactions – Integrate node labels better • Make batch response parsing more efficient – e.g., don't stream if response is not huge • Add traversal functionality • Beautify and deodorize 35

Editor's Notes

  1. Simplified data model for a cancer genomics study
  2. Real (high level) model for a cancer genomics study. Think of every node as representing a table, and every edge as a foreign key (and potentially a linking table). Imagine the join you would have to write to find records on the far left that are related to records on the far lower right. Because of that complexity, you would probably not build the structure you see here in a RDBMS. But then your model is serving the technology at the expense of representing the real world relationships and items.
  3. the batch {} sugar indicates that the calls that would have been made immediately are deferred and kept in a queue, to be emitted after the code inside is done. 'discard_objs' means don't preserve the object information in memory.
  4. From the connect() method of REST::Neo4p::Agent.
  5. This is a little against the NoSQL and graph grain. But the fact is that data stewardship requirements may put you the position of explaining how your apps will positively maintain the integrity and connectivity of the data. Schemas are not dead. If your team is >1 member, there needs to be some externally established and consultable way to know what is in your datastore. If you have a client that wants to make sure you have stored everything she wants stored, you have to be able to report, validate and verify that.
  6. Right thing to do in principle, also not creating thousands of query objects, plus the server tends to bork on thousands of new queries. SomaFM DefCon radio snip: “Devs won’t write parameterized queries unless there’s a gun to their head. We know, we hold the gun.”
  7. The transaction REST endpoint is different from the cypher REST endpoint. Neo4p pays attention to whether you're in a transaction or not, and informs the Agent which endpoint to use. In adding transaction support to Neo4p, identified a bug in 2.0.0. Submitted a ticket and it got fixed! Now, my responsibility not to lead users down the rosy path – check server version and throw if user has a server <2.0.1.