A Little Graph Theory for the Busy Developer - Jim Webber @ GraphConnect Chicago 2013
•Download as PPTX, PDF•
6 likes•3,412 views
In this talk we'll explore powerful analytic techniques for graph data. Firstly we'll discover some of the innate properties of (social) graphs from fields like anthropology and sociology. By understanding the forces and tensions within the graph structure and applying some graph theory, we'll be able to predict how the graph will evolve over time. To test just how powerful and accurate graph theory is, we'll also be able to (retrospectively) predict World War 1 based on a social graph and a few simple mechanical rules. Then we'll see how graph matching can be used to extract online business intelligence (for powerful retail recommendations). In turn we'll apply these powerful techniques to modelling domains in Neo4j (a graph database) and show how Neo4j can be used to drive business intelligence. Don't worry, there won't be much maths :-)
Recommended
Recommended
More Related Content
Similar to A Little Graph Theory for the Busy Developer - Jim Webber @ GraphConnect Chicago 2013
Similar to A Little Graph Theory for the Busy Developer - Jim Webber @ GraphConnect Chicago 2013 (20)
More from Neo4j
More from Neo4j (20)
Recently uploaded
Recently uploaded (20)
A Little Graph Theory for the Busy Developer - Jim Webber @ GraphConnect Chicago 2013
- 1. A Little Graph Theory for the Busy Developer Jim Webber Chief Scientist, Neo Technology @jimwebber
- 2. Roadmap • Imprisoned data • Graph models • Graph theory – Local properties, global behaviors – Predictive techniques • Graph matching – Real-time analytics for fun and profit • Fin
- 7. Aggregate-Oriented Data http://martinfowler.com/bliki/AggregateOrientedDatabase.html “There is a significant downside - the whole approach works really well when data access is aligned with the aggregates, but what if you want to look at the data in a different way? Order entry naturally stores orders as aggregates, but analyzing product sales cuts across the aggregate structure. The advantage of not using an aggregate structure in the database is that it allows you to slice and dice your data different ways for different audiences. This is why aggregate-oriented stores talk so much about map-reduce.”
- 9. complexity = f(size, connectedness, uniformity)
- 12. Property graphs • Property graph model: – Nodes with properties – Named, directed relationships with properties – Relationships have exactly one start and end node • Which may be the same node
- 13. stole from loves loves enemy enemy A Good Man Goes to War appeared in appeared in appeared in appeared in Victory of the Daleks appeared in appeared in companion companion enemy
- 14. Property graphs are very whiteboard-friendly
- 16. http://en.wikipedia.org/wiki/File:Leonhard_Euler_2.jpg Meet Leonhard Euler • Swiss mathematician • Inventor of Graph Theory (1736) 16
- 17. Königsberg (Prussia) - 1736
- 18. A B D C
- 22. Triadic Closure name: Kyle name: Stan name: Kenny
- 23. Triadic Closure name: Kyle name: Stan name: Kenny name: Kyle name: Stan name: Kenny FRIEND
- 24. Structural Balance name: Cartman name: Craig name: Tweek
- 25. Structural Balance name: Cartman name: Craig name: Tweek name: Cartman name: Craig name: Tweek FRIEND
- 26. Structural Balance name: Cartman name: Craig name: Tweek name: Cartman name: Craig name: Tweek ENEMY
- 27. Structural Balance name: Kyle name: Stan name: Kenny name: Kyle name: Stan name: Kenny FRIEND
- 28. Structural Balance is a key predictive technique And it’s domain-agnostic
- 29. Allies and Enemies UK GermanyFrance Russia Italy Austria
- 30. Allies and Enemies UK GermanyFrance Russia Italy Austria
- 31. Allies and Enemies UK GermanyFrance Russia Italy Austria
- 32. Allies and Enemies UK GermanyFrance Russia Italy Austria
- 33. Allies and Enemies UK GermanyFrance Russia Italy Austria
- 34. Allies and Enemies UK GermanyFrance Russia Italy Austria
- 35. Predicting WWI [Easley and Kleinberg]
- 36. Strong Triadic Closure It if a node has strong relationships to two neighbours, then these neighbours must have at least a weak relationship between them. [Wikipedia]
- 37. Triadic Closure (weak relationship) name: Kenny name: Stan name: Cartman
- 38. Triadic Closure (weak relationship) name: Kenny name: Stan name: Cartman name: Kenny name: Stan name: Cartman FRIEND 50%
- 39. Weak relationships • Relationships can have “strength” as well as intent – Think: weighting on a relationship in a property graph • Weak links play another super-important structural role in graph theory – They bridge neighbourhoods
- 40. Local Bridges FRIEND name: Kenny name: Stanname: Kyle FRIEND FRIEND name: Sally name: Bebename: Wendy FRIEND FRIEND 50% name: Cartman FRIEND ENEMY
- 41. Local Bridge Property “If a node A in a network satisfies the Strong Triadic Closure Property and is involved in at least two strong relationships, then any local bridge it is involved in must be a weak relationship.” [Easley and Kleinberg]
- 43. Graph Partitioning • (NP) Hard problem – Recursively remove the spanning links between dense regions – Or recursively merge nodes into ever larger “subgraph” nodes – Choose your algorithm carefully – some are better than others for a given domain • Can use to (almost exactly) predict the break up of the karate club!
- 44. University Karate Clubs (predicted by Graph Theory) 9
- 45. University Karate Clubs (what actually happened!)
- 47. Cypher • Declarative graph pattern matching language – “SQL for graphs” – Columnar results • Supports graph matching commands and queries – Find me stuff like this… – Aggregation, ordering and limit, etc.
- 49. Firstname: Mickey Surname: Smith DoB: 19781006 SKU: 5e175641 Product: Badgers Nadgers Ale SKU: 2555f258 Product: Peewee Pilsner Category: beer SKU: 49d102bc Product: Baby Dry Nights Category: nappies Category: baby Category: alcoholic drinks SKU: 49d102bc Product: XBox 360 Category: consumer electronics Category: console BOUGHTBOUGHT MEMBER_OF MEMBER_OFMEMBER_OF MEMBER_OFMEMBER_OF
- 51. Firstname: * Surname: * DoB: 1996 > x > 1972 Category: beerCategory: nappies BOUGHTCategory: game console
- 52. Firstname: * Surname: * DoB: 1996 > x > 1972 Category: beerCategory: nappies !BOUGHTCategory: game console
- 55. Wrap in a Cypher MATCH clause MATCH (daddy)-[:BOUGHT]->()-[:MEMBER_OF]->(nappies), (daddy)-[:BOUGHT]->()-[:MEMBER_OF]->(beer), (daddy)-[b:BOUGHT]->()-[:MEMBER_OF]->(console)
- 56. Cypher WHERE clause MATCH (daddy)-[:BOUGHT]->()-[:MEMBER_OF]->(nappies), (daddy)-[:BOUGHT]->()-[:MEMBER_OF]->(beer), (daddy)-[b:BOUGHT]->()-[:MEMBER_OF]->(console) WHERE b is null
- 57. Full Cypher query START beer=node:categories(category=‘beer’), nappies=node:categories(category=‘nappies’), xbox=node:products(product=‘xbox 360’) MATCH (daddy)-[:BOUGHT]->()-[:MEMBER_OF]->(beer), (daddy)-[:BOUGHT]->()-[:MEMBER_OF]->(nappies), (daddy)-[b?:BOUGHT]->(xbox) WHERE b is null RETURN distinct daddy
- 58. Results ==> +---------------------------------------------+ ==> | daddy | ==> +---------------------------------------------+ ==> | Node[15]{name:"Rory Williams",dob:19880121} | ==> +---------------------------------------------+ ==> 1 row ==> 0 ms ==> neo4j-sh (0)$
- 60. Facebook Graph Search Which sushi restaurants in NYC do my friends like?
- 61. Graph Structure
- 62. Cypher query START me=node:person(name = 'Jim'), location=node:location(location='New York'), cuisine=node:cuisine(cuisine='Sushi') MATCH (me)-[:IS_FRIEND_OF]->(friend)-[:LIKES]->(restaurant) -[:LOCATED_IN]->(location),(restaurant)-[:SERVES]->(cuisine) RETURN restaurant
- 63. Search structure
- 65. What are graphs good for? • Recommendations • Pharmacology • Business intelligence • Social computing • Geospatial • MDM • Data center management • Web of things • Genealogy • Time series data • Product catalogue • Web analytics • Scientific computing • Indexing your slow RDBMS • And much more!
- 66. Free O’Reilly book for everyone! http://graphdatabases.com
- 67. Thanks for listening Neo4j: http://neo4j.org Me: @jimwebber
Editor's Notes
- Vanity slide - Work for Neo Technology, the commercial backer of the Neo4j open source graph database
- We’ve all been there – trudging through 3 intermediate tables just to get the data you want.
- We have key-value stores, typically very highly available and scalable for simple key-value data
- Column stores naturally-indexed value storesContrary to common belief – Google’s big table isn’t the world’s most famous column storeBritish museum London is: it’s got columns and it’s where we stored all the stuff we nicked from the British Empire!
- Aggregates:No more impedance mismatch, just decompose to documents/ k-v / columnsThe data model is less expressive than RDBMS, but that’s OK because:RDBMS constraints don’t really match application concernsThe expressiveness is in your domain logic, right?
- People talk about Codd’s relational model being mature because it was proposed in 1969 – 42 years old.Euler’s graph theory was proposed in 1736 – 275 years old.
- Graphs are the most natural way to model most domains. You already know this because you draw graphs on a whiteboard, but you’ve never had the opportunity to take that down into the database before.Nodes are a bit like documents, but they’re flat at present in Neo4j.You pour data into your nodes and then connect them – easy peasy.This enables high fidelity domain modeling because this is how your domains work.And you don’t have to do this stuff in your application code – it’s right there in the databaseLet’s prove it by exploring a fun domain…
- Start off sketching the domain. That’s your model done – we see this when we revisit databases months after they’re been designed and put into productionNo decomposition, ER design, normalisation/denormalisation as you need with RDBMS.
- Predictive analytics
- Euler reduced the problem into an abstract formFrom geography to diagrammatic through to a graph.It’s the first documented case of representing the real world as a graph.
- First we need to talk about some local properties
- A triadic closure is a local property of (social) graphs whereby if two nodes are connected via a path involving a third node, there is an increased likelihood that the two nodes will become directly connected in future. This is a familiar enough situation for us in a social setting whereby if we happen to be friends with two people, ultimately there's an increased chance that those people will become direct friends too, since by being our friend in the first place, it's an indication of social similarity and suitability. It’s called triadic closure, because we try to close the triangle.
- We see this all the time – it’s likely that if we have two friends, that they will also become at least acquaintances and potentially friends themselves!In general, if a node A has relationships to B & C then the relationship between B&C is likely to form – especially if the existing relationships are both strong.This is an incredibly strong assertion and will not be typically upheld by all subgraphs in a graph. Nonetheless it is sufficiently commonplace (particularly in social networks) to be trusted as a predictive aid.
- Sentiment plays a role in how closures form too – there is a notion of balance.
- From a triadic closure perspective this is OK, but intuitively it seems odd.Cartman’s friends shouldn’t be friends with his enemies. Nor should Cartman’s enemies be friends with his friends.
- This makes sense – Cartman’s friend Craig is also an enemy of Cartman’s enemy TweekTwo negative sentiments and one positive sentiment is a balanced structure – and it makes sense too since we gang up with our friends on our poor beleaguered enemy
- Another balanced – and more pleasant – arrangement is for three positive sentiments, in this case mutual friends.
- A starting point for a network of friends and enemiesRed links indicate enemy of relationshipBlack links indicate friend of relationshipThe Three Emperor’s league
- Italy forms the with Austria and Germany – a balanced +++ triadic closureIf Italy had made only a single alliance (or enemy) it would have been unstable and another relationship would be likely to form anyway!Triple Alliance
- Russia becomes hostile to Austria and Germany – a balance --+ d triadic closure becomes agnostic towards France.German-Russian Lapse
- The French and Russians ally, forming a balanced --+ triadic closure with the UKFrench-Russian Alliance
- The UK and France enter into the famous Entente CordialeThis produces an unbalanced ++- triadic closure with Russia, and the graph doesn’t like it.
- The British and Russians form an alliance, thereby changing their previously unbalanced triadic closure into a balanced one.Other local pressures on the graph make other closures form.Italy becomes hostile to Russia, forming a balanced --+ closure with the France, and another balanced --+ closure with the UK.Germany and the UK become hostile forming a balanced --+ closure with Austria and another balanced --+ closure with ItalyBritish-Russian Alliance
- That WWI can be predicted without domain knowledge by iterating a graph and applying local structural constraints is nothing short of astonishing to me.Note how the network slides into a balanced labeling — and into World War I.
- In this case the string triadic closure property still holds – though it is a weak link that characterises the relationship between Stan and Cartman.Given a starting graph, we can apply this simple local principal to see how it would evolve.
- In this case the string triadic closure property still holds – though it is a weak link that characterises the relationship between Stan and Cartman.Given a starting graph, we can apply this simple local principal to see how it would evolve.
- A local bridge acts as a link – perhaps the only realistic link - between two otherwise distant (or separate) subgraphs.Local bridges are semantically rich – they provide conduits for information flow between otherwise independent groups.In this case DATING is a local bridge – it must also be a weak relationship according to our definition of a local bridgeIntuitively this makes sense – your girl/boyfriend is rather less important at age 8 than your regular friends, IIRC.
- How do we identify local bridges? Any weak link which would cause a component of the graph to become disconnected.Being able to identify local bridges is important – in this case it’s the only know conduit to allow the girls and boys to communicate.In real life local bridges are apparent in your organisation as experts (or managers); appear as nexus in fraud cases;
- Zachary in the Journal of Anthropological Research 1977Intuitively we can see “clumps” in this graph.But how do we separate them out? It’s called minimum cut.
- What’s interesting is that it’s mechanical – no domain knowledge is necessary. There’s only one failure with the method Zachary chose to partition the graph: node 9 should have gone to the instructor’s club but instead went with the original president of the club (node 34).Why? Because the student was three weeks away from completing a four-year quest to obtain a black belt, which he could only do with the instructor (node 1)Other minimum cut approaches might deliver slightly different results, but on the whole it’s amazing you get such insight from an algorithm!
- But is there enough information in the graph itself to predict the schism?
- But is there enough information in the graph itself to predict the schism?
- We can use graph matching to look for behavioural patterns in the graph too!
- The insight here is that we have a typical young father who buys beer, nappies and a game console simply by reducing subgraphWe have a pattern to search for
- Now we look for young fathers – implied by beer and nappies purchases – who haven’t bought a game console.
- START n=node(*), r=rel(*) DELETE r,nCREATE daddy1 = { name: 'Mickey Smith', dob: 19781006 }CREATE beer = { category: 'beer' } CREATE alcohol = { category: 'alcoholicdrinks' } CREATE peeweePilsner = { sku: '2555f258', product: 'PeeweePilsner' } CREATE badgersNadgers = { sku: '5e175641', product: 'BadgersNadgers Ale' }CREATE peeweePilsner-[:MEMBER_OF]->beerCREATE badgersNadgers-[:MEMBER_OF]->beerCREATE beer-[:MEMBER_OF]->alcoholCREATE daddy1-[:BOUGHT]->peeweePilsnerCREATE daddy1-[:BOUGHT]->badgersNadgersCREATE baby = { category: 'baby' }CREATE nappies = { category: 'nappies' }CREATE nappies-[:MEMBER_OF]->babyCREATE babyDryNights = { sku: '49d102bc', product: 'Baby DryNights'} CREATE babyDryNights-[:MEMBER_OF]-nappiesCREATE daddy1-[:BOUGHT]->babyDryNightsCREATE consumerElectronics = { category: 'consumerelectronics' }CREATE console = { category: 'console' }CREATE xbox = { sku: '49d102bc', product: 'XBox 360' }CREATE xbox-[:MEMBER_OF]->(console)-[:MEMBER_OF]->consumerElectronicsCREATE daddy1-[:BOUGHT]->xboxCREATE mummy11 = { name: 'Rose Tyler', dob: 19800317 }CREATE wine = { sku:'3a3f22bc', product: 'Shiraz' }CREATE mummy1-[:BOUGHT]->wineCREATE mummy1-[:BOUGHT]->babyDryNights CREATE daddy2 = { name: 'Rory Williams', dob: 19880121 }CREATE daddy2-[:BOUGHT]->peeweePilsnerCREATE daddy2-[:BOUGHT]->babyDryNightsSTART beer=node(2), nappies=node(7), xbox=node(11)MATCH (daddy)-[:BOUGHT]->()-[:MEMBER_OF]->(beer), (daddy)-[:BOUGHT]->()-[:MEMBER_OF]->(nappies), (daddy)-[b?:BOUGHT]->(xbox)WHERE b isnullRETURN distinctdaddy
- Note that Max de Marzireimplemented a functionally better Graph Search with Neo4j and some Ruby gems for language processing in a weekend!
- Image: real-timeBeing able to look for all young fathers who might be tempted to buy a new game console is helpful, but not dramatically different from what we have nowIt’s much faster to process in a graph, but still a latent business activity (e.g. mailshot)But you can take the same idea and run the query in real time foran individual customer at they go through the checkout!And you can do more too – you can add in more/less dimensions to the search.Does the shop have stock?Does the young father live in the right target area (r-tree, postcode)?We can vary the number of dimensions we include to tailor the search for performance/accuracy very easily in a graph – query latency is proportional to the amount of graph searched, not data set size.