Neo4j -[:LOVES]-> Cypher

Geekout Tallin EstoniaGot a Graph Database? Need a Query Language! (Neo4j) -[:LOVES]-> (Cypher)

(Michael) -[:WORKS_ON]-> (Neo4j) consoleCypher community graph Community MEServer Spring Cloud

YOU ? SQL NOSQLGraph Database

Graphs areeverywhere

(Neo4j) -[:IS_A]-> (Graph Database) Lucene Sharding 1 M/s Master/ Index Slave LS TRAVERSA HIG S H_A TE VA RA IL. TEG IN PROVIDES Server ACID TX RUN LI S_A CE S NS ED O _L Ruby IK ES_T RU JS E MySQL S NS _A _O SC ALClojure NS .net N RU Mongo embedded 34bn Heroku Java Nodes

Homework- so Pay Attention• Go To http://bit.ly/geekout-cypher• With your Cypher knowledge • Add yourself to the graph • Determine a path from you to me• Share the console on Twitter by Monday• Win this AR-Drone

Query a Graph with ___ _____ _____ _____ _____ ___ ______ _____ / _ / ___/ __ _ _|_ _| / _ | ___ _ _|/ /_ `--.| / / | | | | / /_ |_/ / | || _ |`--. | | | | | | _ | / | || | | /__/ / __/_| |_ _| |_ | | | | | | |_| |_|____/ ____/___/ ___/ _| |_|_| _| _/Whirlwind Tour

A Graph Can‘t see the Patterns for the Trees http://maxdemarzi.com/2012/02/13/visualizing-a-network-with-cypher/

Patterns? Where?

Patterns? Describe them!

Patterns? Find them on bound nodes

Patterns? There are more !

Patterns? Make them visible!

How ?A Graph Query Language with Pattern Matching

SparQL, SQL ? No, Cypherdeveloped by and for Neo4j

Can I draw Patterns? A B C

Sure, with ASCII ART! A B C

Sure, with ASCII ART! A B C A --> B --> C, A --> C

Sure, with ASCII ART! A B C A --> B --> C, A --> C A --> B --> C <-- A

Example Directed Relationship A B

Example Directed Relationship A B (A) --> (B)

ExampleLabeled Directed Relationship LOVES A B

ExampleLabeled Directed Relationship LOVES A B (A) -[:LOVES]-> (B)

ExampleTransitive Relationship Path A B C

ExampleTransitive Relationship Path A B C(A)-->(B)-->(C)

ExampleVariable Length Path A B A B A B ...

ExampleVariable Length Path A B A B A B ... (A) -[*]-> (B)

Real World Examples

Give me: JOINSSELECT skills.*, user_skill.*FROM usersJOIN user_skill ON users.id = user_skill.user_idJOIN skills ON user_skill.skill_id = skill.idWHERE users.id = 1 START user = node(1) MATCH user -[r:USER_SKILL]-> skill RETURN skill, r

Give me: Old, Influential FriendsSTART me = node(...)MATCH (me) - [f:FRIEND] - (old_friend) - [:FRIEND ] - (fof)WHERE ({today}-f.begin) > 365*10WITH old_friend, collect(fof.name) as namesWHERE length(names) > 100RETURN old_friend, namesORDER BY old_friend.name ASC f:FRIEND :FRIEND me friend fof

Give me: Simple RecommendationSTART me = node(...)MATCH (me) -[r1:RATED ]->(thing) <-[r2:RATED ]- (someone) -[r3:RATED ]->(cool_thing)WHERE ABS(r1.stars-r2.stars) <= 2 AND r3.stars > 3RETURN cool_thing, count(*) AS cntORDER BY cnt DESC LIMIT 10 r1:RATED thing r2:RATED me so r3:RATED cool thing

Results ? • Tables: for Human Brainz & Tools • Graphs: to highlight, visualize, export & refining queries

One Step Back• Goals / Intent• Origins• Decisions• Implementation• Future

What is Cypher?• Graph Query Language for Neo4j• Querying for Humans

Goals, Origins & Design Picking good ideas and having some of our own

Cypher: Some Goals ASCII-art Pattern patterns Matching Declarative External DSL Closures SQL Familiarity

Something new? • Existing Neo4j query mechanisms were not simple enough • Too verbose (Java API) • Too prescriptive (Gremlin) • Other query languages

Java API? • Object oriented • Node, Relationship, Path objects • Imperative • Verbose • Traversers, for-loops, Index • mostly lazy-eval

Gremlin? • DSL for pipes • imperative • a single path expression • loop constructs • side-effects • lots of closures

SQL? • Well known and understood • Unable to express paths • these are crucial for graph-based reasoning • Cumbersome Mutation • Neo4j is schema/table free

SPARQL? • SPARQL designed for a different data model • namespaces / URI‘s • reiﬁed properties as nodes • SPARQL/RDF mostly in academia • developers don‘t get it • Pattern matching is cool

Design Decisions Closures / Quantiﬁers START london = node(1), moscow = node(2) MATCH path = london -[*]-> moscow WHERE all(city in nodes(path) where city.capital) RETURN path

Design Decisions Parsed, not an internal DSL Execution Semantics Serialisation Type System Portability

Design Decisions Familiar for SQL users select start from match where where group by return order by order by skip limit

Design Decisions Database vs Application Design Goal: single user interaction expressible as single query Queries have enough logic to ﬁnd required data, not enough to process it

Implementation Docs fromExecution Tests Java Bridge Plan Pattern Matching Parsing Scala

Scala• Parser Combinator• Query Object Creation• Combining Pipes• Pattern Matching• Lazy Evaluation

Example: PipesSTART n=node(0)MATCH n-[*]->mRETURN n, count(*)ORDER BY n.nameResulting Pipes:Parameters()Nodes(n) // Identifier nPatternMatch(n-[*]->m) // Identifier mExtract([n,n.name])Sort(n.name ASC)EagerAggregation( keys: [n], aggregates: [count(*)])ColumnFilter([n.name,n,count(*)])

Why Scala?• Fun Language• Parser Combinators• Functional Concepts• Collection library• Immutable Values• Scala Pattern Matching

Problems with Scala?• Scala versions• Slow compilation• Low Collection Write Throughput• Code gets easily complicated• Hard to ramp up for other devs• Big separate library

Implementation Query Parts• START• MATCH • Pattern Matching• WHERE • Expressions, Predicates• RETURN

START WHERE Aggregation BINDS FILTERSDELETE RESTRUCTURES IdentiﬁerREMOVE_FROM Result BOUND_TO USED_IN PAGINATES BUILDS_UP SKIP/LIMIT Graph FOUND_IN Pattern DESCRIBES ADD_TO CREATES MATCH FIX COMPLETES CREATE RELATE

STARTSTART dev=node:user(name=“Andres“) • lazy source of identiﬁers bound to nodes and relationships • each identiﬁer is an iterable • spawns execution per value • cross product between multiple • index lookup or direct lookup

MATCHMATCH dev-[:WORKED_ON]->project• describe patterns with ASCII art• declare identiﬁers • paths • var. length paths • graph algorithms • optional relationships• expands results with each found subgraph

Core Acitvity: Pattern Matching• Match clause with ASCII-art• derive pattern description• bound Nodes and Relationships• ﬁnds patterns attached to bound entities• each found Pattern spawns Subgraph Result• recursive PM with Backtracking

Pattern Matching• Scala • incremental search with backtracking • allows: • Variable length paths • Filter during matching • optional relationships • powerful but slower

Pattern Matching• Java • existing graph matching libray • fast but less capable • integrated with lazy Scala API• NEW Traversal Framework approach • for one or two bound nodes • use Neo4j traversal framework• pattern description & complexity determine Pattern Matcher selection

Implementation • Recursive matching with backtrackingSTART x=... MATCH x-->y, x-->z, y-->z, z-->a-->b, z-->b

WHEREWHERE project.name = „Cypher“• ﬁlters results• single big boolean expression• needs existing identiﬁers to work with• much like SQL

Expressions• expressions compute values • composite Speciﬁcation Pattern • input is ExecutionContext Map• have name• declare symbolic dependencies • self & composite• directly derived from parser • probably rewritten in between

Predicates• WHERE clause ﬁlters results with a single composed predicate• boolean expressions • composable boolean algebra• Patterns as predicates• Quantiﬁers (ALL, NONE, SINGLE)• Collections as predicates

RETURNRETURN project, collect(idea),count(fun) • determines what to return • column names or aliases with AS • automatic aggregation • when aggregation function exists • all non-aggregated values are grouping • lazyness is killed with aggregation

SKIP LIMIT ORDER BY SKIP 5 LIMIT 10 ORDER by length(ideas) DESC • the usual suspects • lazyness is killed with ordering

Mutation Transaction WITH Idem-(Separation) potence DRY Mass Data Handling Lazyness

Mutation• need transactions• must not disturb reads / traversals • separation of read and write query parts• explicit context change and scope: WITH • implicit split for simple queries• granularity of mutation aligned with # of executions• work with parameters

Mutation - Impl.• Query Builder create UpdateCommands• UpdatePipe ??• Transaction Scope for whole Query• collect statistics• create new entities • add identiﬁers to the ExecutionContext• need to iterate through to get all updates executed (no lazyness), special Pipe• have to track already deleted entities

ImplementationMutating Query Parts• WITH• CREATE• RELATE• SET, DELETE• FOREACH

START WHERE Aggregation BINDS FILTERSDELETE RESTRUCTURES IdentiﬁerREMOVE_FROM Result BOUND_TO USED_IN PAGINATES BUILDS_UP SKIP/LIMIT Graph FOUND_IN Pattern DESCRIBES ADD_TO CREATES MATCH FIX COMPLETES CREATE RELATE

WITHWITH me, count(friend) as friends • syntax like RETURN • separate query parts like a pipe • declares a new scope with new identiﬁers • all other ids will be gone • useful for HAVING • spawns a new ExecutionContext • continue with read or write part

CREATECREATE me = {name: „Michael“}• create new nodes or relationships• can work with map params (or Iterables thereof)• assign new identiﬁers• can create full paths

RELATERELATE posts-[:POST]->(p {title: „..“} • FIX the graph • construct missing relationships and nodes • need at least one bound node • match given properties • try to advance (from multiple sides) and create missing stuff then iterate

DELETE DELETE n, rel, m.prop• as expected• idempotent deletion• can only delete unconnected nodes • so delete relationships ﬁrst START n=node(*) MATCH n-[r?]-() DELETE n,r

SETSET n.name = „Father of “+m.name • can work with arbitrary expressions • use coalesce for idempotent defaults • can mass assign with map-parameters

FOREACHFOREACH ( f in new_friends : RELATE me-[:FRIEND]->f) • iterable loop for mutating operations • saves a lot of repetetive code • wraps current execution context in a temporary proxy

Self Documenting Tests• Provides • Title & Description • Deﬁne Sample Graph • Declare & Execute Query • Results

Self Documenting Tests• Asserts • No Syntax Errors • Multiple Cypher Versions • No Execution Errors • Results • Resulting graph

Self Documenting Tests• Generates • Graph Rendering Graph-Viz • Tabular Results • Ascii-doc for Documentation • Live-Console Integration

Self Documenting Tests • Good enough for • Fun to write • Integration Tests • Manual • Blog-ready Cookbook Examples

Self DocumentingTests

Cypher Console(s)• Integrated in Neo4j-Shell• Integrated in Webadmin (Shell & Search)• REPL & GIST - console.neo4j.org• Interactive learning GCLI• Scala spray-can & cypher = bansky• py2neo, bash• Google Document

Live Console• In Memory GDBs in Web Session• Set up with mutating Cypher (or Geoff)• Executes Cypher (also mutating)• Visualizes Graph & Query Results (d3)• Multiple Cypher Versions• Share: short link, tweet, yUML• Embeddable <iframe>• Live Console for Docs

The Rabbithole http://console.neo4j.org This Graph: http://tinyurl.com/7cnvmlq

Homework- Apply your Wits• Go To http://bit.ly/geekout-cypher• With your new Cypher knowledge • Add yourself to the graph • Determine a path from you to me• Share the console on Twitter by Monday• Win this AR-Drone

Graph your World• Neo4j.org• Google Group• Grab your own ego-Graph

Thanks for Listening! Questions?michael.hunger@neotechnology.com @mesirii

Neo4j -[:LOVES]-> Cypher

by jexp on Jun 15, 2012

Accessibility

Categories

Upload Details

Usage Rights

1 Embed 138

Statistics

Neo4j -[:LOVES]-> Cypher Presentation Transcript