Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Next generation Polyglot Architectures using Neo4j by Stefan Kolmar

414 views

Published on

https://www.bigdataspain.org/2016/program/thu-next-generation-polyglot-architectures-using-neo4j.html

https://www.youtube.com/watch?v=kVbUJHNyccc&t=18s&index=32&list=PL6O3g23-p8Tr5eqnIIPdBD_8eE5JBDBik

Published in: Technology
  • Sie können Hilfe bekommen bei ⇒ www.WritersHilfe.com ⇐. Erfolg und Grüße!
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Be the first to like this

Next generation Polyglot Architectures using Neo4j by Stefan Kolmar

  1. 1. Neo Technology Polyglot Architectures September 2016 Stefan Kolmar Director Field Engineering
  2. 2. Agenda •  Drivers for change in enterprises •  Requirements created for polyglot architecture •  Tradi<onal approaches •  Benefits of using graph technology •  Neo4j as na<ve GraphDB •  Architectures and Ecosystems •  Case studies •  Conclusions Admin
  3. 3. Drivers for change in enterprises • Data explosion Admin
  4. 4. Drivers for change in enterprises • Data explosion • IoT / Industrie 4.0 Admin
  5. 5. Drivers for change in enterprises • Data explosion • IoT / Industrie 4.0 • Cloud Compu<ng Admin
  6. 6. Drivers for change in enterprises • Data explosion • IoT / Industrie 4.0 • Cloud Compu<ng • Need to use Connected Data Admin
  7. 7. The need of connected data in the Enterprise Connected Enterprise Employees, Customers and Partners Digital Mesh Internet of Connected Things Knowledge Network 7
  8. 8. Requirements created for polyglot architecture •  Ability to handle large scale data volumes •  High availability 24x7 •  Open interfaces •  Structure data to take advantage of connec<ons •  Ability to query connected data on demand •  Flexibility to connect and query data with different schema -> Using the technology which fits best doing the job Admin
  9. 9. TradiBonal approaches •  Using RDBMS systems: •  En<ty rela<onship models •  Logical and physical Schema •  SQL queries Admin
  10. 10. Database Landscape 1990 RDBMS 2000 RDBMS (Opera<onal) Data Warehouse 2010 RDBMS Hadoop NoSQL OLAP/DW Today RDBMS Hadoop NoSQL OLAP/DW GraphDB Spark
  11. 11. Today: sBll the reality storing data
  12. 12. What is the problem with that?
  13. 13. What is the problem with that? •  Understandable for business users? •  Agility? •  Needs effort to materialize rela<onships •  Logical versus physical data model? •  Op<miza<on needed to perform queries?
  14. 14. Case Studies
  15. 15. What is the most powerful database in the world?
  16. 16. AlternaBve Database: Graph Data Model
  17. 17. Advantages using Graph DB Time to market Flexibility Opera<onal Efficiency
  18. 18. The Whiteboard Model Is the Physical Model
  19. 19. CAR name: “Dan” born: May 29, 1960 twifer: “@dan” name: “Ann” born: Dec 5, 1995 since: Jan 10, 2011 brand: “Volvo” model: “V70” Property Graph Model Components Nodes •  The objects in the graph •  Can have name-value proper@es •  Can be labeled RelaBonships •  Relate nodes by type and direc<on •  Can have name-value proper@es FATHER OF LIVES WITH LIVES WITH PERSON PERSON
  20. 20. RelaBonal Versus Graph Models RelaBonal Model Graph Model KNOWS ANDREAS TOBIAS MICA DELIA Person Friend Person-Friend ANDREAS DELIA TOBIAS MICA
  21. 21. Cypher: Powerful and Expressive Query Language KNOWS Dan Ann MATCH (:Person { name:“Dan”} ) -[:KNOWS]-> (:Person { name:“Ann”} ) LABEL PROPERTY NODE NODE LABEL PROPERTY
  22. 22. MATCH (boss)-[:MANAGES*0..3]->(sub), (sub)-[:MANAGES*1..3]->(report) WHERE boss.name = “John Doe” RETURN sub.name AS Subordinate, count(report) AS Total Express Complex Queries Easily with Cypher Find all direct reports and how many people they manage, up to 3 levels down Cypher Query SQL Query
  23. 23. Connectedness and Size of Data Set Response Time 0 to 2 hops 0 to 3 degrees Thousands of connec<ons Tens to hundreds of hops Thousands of degrees Billions of connec<ons Rela<onal and Other NoSQL Databases Neo4j Neo4j is 1000x faster Reduces minutes to milliseconds Benchmark real-Bme query performance Neo4j Vs Rela<onal and other NoSQL Databases
  24. 24. Advantages using Neo4j Graph DB Time to market Flexibility Opera<onal Efficiency Query speed / Performance for connected data Schema free Traversing vs Joins Cypher query language
  25. 25. What is a NaBve Graph Database? Applica<on MATCH (:Person { name:“Dan”} ) -[:KNOWS]-> (:Person { name:“Ann”} ) Processing Engine
  26. 26. Agenda •  Drivers for change in enterprises •  Requirements created for polyglot architecture •  Tradi<onal approaches •  Benefits of using graph technology •  Neo4j as na<ve GraphDB •  Architectures and Ecosystems •  Case studies •  Conclusions Admin
  27. 27. PaTerns in Neo4j architectures •  Primary Store •  Reads + Write •  Secondary Store •  Mostly read, scheduled batch updates •  ODS / MDS •  Mostly read, scheduled batch updates + integra<on logic
  28. 28. Primary Store case: Telenor •  Iden<ty and access management problem. Need to compute resource authoriza<on in real <me Was taking up to 20 min for large customers -> precalcula<on & cache -> stale data
  29. 29. Original architecture 500 requests / sec Middleware services Backend Backend Backend Channel Channel Channel 42 channels 35 systems
  30. 30. New architecture 500 requests / sec Middleware services Backend Backend Backend Channel Channel Channel 42 channels 35 systems
  31. 31. Did it work? •  Response <mes reduced to seconds and milliseconds •  Code maintainability improved. Access rules in Cypher as graph paferns. "The Neo4j graph database gives us dras@cally improved performance and a simple language to query our connected data" – Sebas<an Verheughe, Architect & Developer
  32. 32. Secondary Store Use Case : Shutl/eBay Wanted to come up with quotes for delivery slots. It took 2 seconds to come up with a quote for a single delivery slot and they wanted to extend the service to show 20 poten<al delivery slots Original architecture was a monolith based on rails
  33. 33. Original architecture Requests quote Rails monolith Relational
  34. 34. SoluBon Broke applica<on up into services e.g. quo<ng, booking, feedback. Key goal was to improve the speed of quo<ng and neo4j backed this service
  35. 35. New architecture Requests quote Quo<ng service Relational Booking service Feedback service Nightly job to populate ETL
  36. 36. Did it work? •  Quo<ng for 20 <me windows down from 82,000 ms to 80ms •  Code complexity much reduced “We found Neo4j to be literally thousands of 0mes faster than our prior MySQL solu<on, with queries that require 10-100 <mes less code. Today, Neo4j provides eBay with func<onality that was previously impossible.” Volker Pacher, Senior Developer, eBay
  37. 37. Who else uses Neo4j as a data source to feed mission criBcal applicaBons? •  World’s largest parcel service •  They no<ced that they would not have been able to deliver packages by Christmas 2013 •  Hierarchical rou<ng system (RDBMS backed) •  Replaced it with Neo4j using lateral rou<ng •  Value: •  Minimized <me •  Maximized usage of road network •  Recommenda<on engines •  Fraud detec<on / preven<on architectures
  38. 38. Neo4j Web App Data Integra<on RDBMS (Oracle, MySQL, DB2, HANA …) Management Console (E.g BI Tools such as Tableau, Qlik, BO, MicroStrategy etc) Fraud Analyst Machine2Machine generated ac<ons Alert Incoming Events CRM System Opera<onal System External Data Sample Architecture for Fraud DetecBon
  39. 39. Sample Architecture for Internet Retail RecommendaBons Neo4j Internet Shop Data Integra<on Invoicing Shipping etc User Opera<onal System External Data Current Profiles
  40. 40. ODS / MDS case: Gov. Agency •  The graph integrates data from mul<ple sources •  ODS: Addi<onal opera<ons on the integrated data •  MDS: Provide reference data to other opera<onal systems •  European government agencies
  41. 41. TradiBonal ODS/MDS architecture Relational ODS Data Source 1 Data Source 2 Semi / unstructured data source ETL Opera<onal BI Relational MDS EDW Opera<onal Systems
  42. 42. Graph based ODS/MDS architecture ETL Real <me connected data analysis and explora<on EDW Opera<onal Systems Data Source 1 Data Source 2 Semi / unstructured data source
  43. 43. Common Insurance Scenario Internal users Data Layer Business Logic - Data Consumers are employees or Sales agents - Built on Mainframes - Rich Business Logic - During Office Hours suppor<ng OLTP queries - Nightly batch runs - No real <me support 24x7 availability Business Logic Suppor<ng RT Data Layer “Yesterday” “Today”
  44. 44. IoT Touchpoints Device Management Support Devices Provision Devices Security / Access Mgmt Grant Access Request Access MDM Layer Supplier Master Customer Master Product Master Support Master … Revenue Sharing
  45. 45. Conclusions: •  Polyglot architectures are required to sa<sfy the needs exis<ng and coming •  Neo4j na<ve graph database provides a compete<ve edge: •  Agility •  Flexibility •  Opera<onal Efficiency •  Time to Market •  More informa<on: •  www.neo4j.com •  Stefan.Kolmar@neo4j.com
  46. 46. Neo Technology The Graph Database Leader September 2016 Stefan Kolmar Director Field Engineering

×