Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

NoSQL meets Microservices

2,254 views

Published on

Just a few years ago all software systems were designed to be monoliths running on a single big and powerful machine. But nowadays most companies desire to scale out instead of scaling up, because it is much easier to buy or rent a large cluster of commodity hardware then to get a single machine that is powerful enough. In the database area scaling out is realized by utilizing a combination of polyglot persistence and sharding of data. On the application level scaling out is realized by microservices. In this talk I will briefly introduce the concepts and ideas of microservices and discuss their benefits and drawbacks. Afterwards I will focus on the point of intersection of a microservice based application talking to one or many NoSQL databases. We will try and find answers to these questions: Are the differences to a monolithic application? How to scale the whole system properly? What about polyglot persistence? Is there a data-centric way to split microservices?

Published in: Software

NoSQL meets Microservices

  1. 1. www.arangodb.com NoSQL meets Microservices Michael Hackstein @mchacki
  2. 2. Michael Hackstein ‣ ArangoDB Core Team ‣ Web Frontend ‣ Graph visualisation ‣ Graph features ‣ Host of cologne.js ‣ Master’s Degree
 (spec. Databases and
 Information Systems) 2
  3. 3. Classical Setup 3
  4. 4. Scaling 4 LoadBalancer
  5. 5. Responsibilities 5 LoadBalancer Application Code Consistency Joins
  6. 6. Introduction to NoSQL 6 NoSQL World Documents - JSON Graphs Key Value { “type“: "pants", “waist": 32, “length”: 34, “color": "blue", “material”: “cotton" } { “type“: "television", “diagonal screen size": 46, “hdmi inputs": 3, “wall mountable": true, “built-in digital tuner": true, “dynamic contrast ratio”: “50,000:1”, Resolution”: “1920x1080” } { “type": "sweater", “color": "blue", “size": “M”, “material”: “wool”, “form”: “turtleneck" } { “type": "sweater", “color": "blue", “size": “M”, “material”: “wool”, “form”: “turtleneck" } ‣ Map value data to unique string keys (identifiers) ‣ Treat data as opaque (data has no schema) ‣ Can implement scaling and partitioning easily ‣ Focussed on m-to-n relations between entities ‣ Stores property graphs: entities and edges can have attributes ‣ Easily query paths of variable length ‣ Normally based on key-value stores (each document still has a unique key) ‣ Allow to save documents with logical similarity in “collections” ‣ Treat data records as attribute-structured documents (data is no more opaque) ‣ Often allow querying and indexing document attributes
  7. 7. Polyglot Modeling ‣ If you have structured data ➡ Use a document store ‣ If you have relations between entities and want to efficiently query them by arbitrary steps in between ➡ Use a graph database ‣ If you manage the data structure yourself and do not need complex queries ➡ Use a key-value store ‣ If you have structured data in graph format, or data model might change ➡ Use a multi-model database 7 Use the right tool for the job
  8. 8. Using different databases 8 LoadBalancer
  9. 9. Responsibilities 9 LoadBalancer Application Code Consistency Joins
  10. 10. Responsibilities 9 LoadBalancer Application Code Consistency Joins
  11. 11. Solution 1: Writes only to one master 10
  12. 12. Solution 2: Microservices 11
  13. 13. app Monolith breakdown 12 server.route({ method: "GET", path: "/app", handler: function (req, reply) { var sum = cpuWaste(); reply(dataLookup["key" + sum]); } }); var cpuWaste = function() { var sum = Math.floor(Math.random() * 10000); for (var j = 0; j < 1000000; ++j) { sum += j; sum %= 10000; } return sum; }; var dataLookup = {}; for (var j = 0; j < 500000; ++j) { dataLookup["key" + j] = "This is test text number " + j + "."; }
  14. 14. memService cpuService app Monolith breakdown 12 server.route({ method: "GET", path: "/app", handler: function (req, reply) { var sum = cpuWaste(); reply(dataLookup["key" + sum]); } }); var cpuWaste = function() { var sum = Math.floor(Math.random() * 10000); for (var j = 0; j < 1000000; ++j) { sum += j; sum %= 10000; } return sum; }; var dataLookup = {}; for (var j = 0; j < 500000; ++j) { dataLookup["key" + j] = "This is test text number " + j + "."; }
  15. 15. memService cpuService app Monolith breakdown 12 server.route({ method: "GET", path: "/app", handler: function (req, reply) { var sum = cpuWaste(); reply(dataLookup["key" + sum]); } }); var cpuWaste = function() { var sum = Math.floor(Math.random() * 10000); for (var j = 0; j < 1000000; ++j) { sum += j; sum %= 10000; } return sum; }; var dataLookup = {}; for (var j = 0; j < 500000; ++j) { dataLookup["key" + j] = "This is test text number " + j + "."; } server.route({ method: "GET", path: "/mem/{id}", handler: function (request, reply) { reply(dataLookup["key" + request.params.id]); } });
  16. 16. memService cpuService app Monolith breakdown 12 server.route({ method: "GET", path: "/app", handler: function (req, reply) { var sum = cpuWaste(); reply(dataLookup["key" + sum]); } }); var cpuWaste = function() { var sum = Math.floor(Math.random() * 10000); for (var j = 0; j < 1000000; ++j) { sum += j; sum %= 10000; } return sum; }; var dataLookup = {}; for (var j = 0; j < 500000; ++j) { dataLookup["key" + j] = "This is test text number " + j + "."; } server.route({ method: "GET", path: "/mem/{id}", handler: function (request, reply) { reply(dataLookup["key" + request.params.id]); } }); server.route({ method: "GET", path: "/cpu", handler: function (request, reply) { reply(cpuWaste()); } });
  17. 17. memService cpuService app Monolith breakdown 12 var cpuWaste = function() { var sum = Math.floor(Math.random() * 10000); for (var j = 0; j < 1000000; ++j) { sum += j; sum %= 10000; } return sum; }; var dataLookup = {}; for (var j = 0; j < 500000; ++j) { dataLookup["key" + j] = "This is test text number " + j + "."; } server.route({ method: "GET", path: "/mem/{id}", handler: function (request, reply) { reply(dataLookup["key" + request.params.id]); } }); server.route({ method: "GET", path: "/cpu", handler: function (request, reply) { reply(cpuWaste()); } }); server.route({ method: "GET", path: "/app", handler: function (req, reply) { request('http://127.0.0.1:9000/cpu', function (error, response, body) { if (!error && response.statusCode == 200) { request('http://127.0.0.1:9000/mem/' + body, function (error, response, body) { if (!error && response.statusCode == 200) { reply(body); } else { reply({error: error}); } }); } else { reply({error: error}); } }); } });
  18. 18. neo4jService mongoDBService productService Polyglot Setup 13 MATCH (me:Users {id: "' + id + '"}) -[:Friend]-> (:Users) -[:Bought]-> (p) return p.id as id products.find({$or: req.payload})
  19. 19. neo4jService mongoDBService productService Polyglot Setup 13 MATCH (me:Users {id: "' + id + '"}) -[:Friend]-> (:Users) -[:Bought]-> (p) return p.id as id products.find({$or: req.payload})
  20. 20. neo4jService mongoDBService productService Polyglot Setup 13 MATCH (me:Users {id: "' + id + '"}) -[:Friend]-> (:Users) -[:Bought]-> (p) return p.id as id products.find({$or: req.payload})
  21. 21. neo4jService mongoDBService productService Polyglot Setup 13 MATCH (me:Users {id: "' + id + '"}) -[:Friend]-> (:Users) -[:Bought]-> (p) return p.id as id products.find({$or: req.payload})
  22. 22. neo4jService mongoDBService productService Polyglot Setup 13 MATCH (me:Users {id: "' + id + '"}) -[:Friend]-> (:Users) -[:Bought]-> (p) return p.id as id products.find({$or: req.payload})
  23. 23. neo4jService mongoDBService productService Polyglot Setup 13 MATCH (me:Users {id: "' + id + '"}) -[:Friend]-> (:Users) -[:Bought]-> (p) return p.id as id products.find({$or: req.payload})
  24. 24. neo4jService mongoDBService productService Polyglot Setup 13 MATCH (me:Users {id: "' + id + '"}) -[:Friend]-> (:Users) -[:Bought]-> (p) return p.id as id products.find({$or: req.payload})
  25. 25. Live Demo Neo4J + MongoDB 14
  26. 26. Multi Model Database ‣ Can natively store several kinds of data models: ‣ Key-value pairs ‣ Documents ‣ Graphs ‣ Delivers query mechanisms for all data models ‣ ACID transactions ‣ Cross collection joins 15
  27. 27. Using a Multi-Model Database 16 LoadBalancer Application Code Consistency Joins
  28. 28. Using a Multi-Model Database 16 LoadBalancer Application Code Consistency Joins
  29. 29. Foxx ‣ Customized REST API on top of ArangoDB ‣ Microservice framework ‣ Integrate with other microservices ‣ Reuse your Node.js code and NPM modules ‣ Built-in authentication using OAuth2.0 or HTTP-Basic Auth ‣ Operations are encapsulated in the database ‣ low network traffic, direct data access ‣ increases data privacy / (~( ) ) /_/ ( _-----_(@ @) ( / /|/--| V " " " " 17
  30. 30. foxx productService Foxx Setup 18 RETURN GRAPH_NEIGHBORS('ecom', @id, {direction: 'outbound', minDepth: 2, maxDepth: 2, neighborCollectionRestriction: 'Products', includeData: true })
  31. 31. foxx productService Foxx Setup 18 RETURN GRAPH_NEIGHBORS('ecom', @id, {direction: 'outbound', minDepth: 2, maxDepth: 2, neighborCollectionRestriction: 'Products', includeData: true })
  32. 32. foxx productService Foxx Setup 18 RETURN GRAPH_NEIGHBORS('ecom', @id, {direction: 'outbound', minDepth: 2, maxDepth: 2, neighborCollectionRestriction: 'Products', includeData: true })
  33. 33. foxx productService Foxx Setup 18 RETURN GRAPH_NEIGHBORS('ecom', @id, {direction: 'outbound', minDepth: 2, maxDepth: 2, neighborCollectionRestriction: 'Products', includeData: true })
  34. 34. foxx productService Foxx Setup 18 RETURN GRAPH_NEIGHBORS('ecom', @id, {direction: 'outbound', minDepth: 2, maxDepth: 2, neighborCollectionRestriction: 'Products', includeData: true })
  35. 35. Live Demo Foxx 19
  36. 36. ‣ open source and free (Apache 2 license) ‣ sharding & replication ‣ JavaScript throughout (V8 built into server) ‣ drivers for a wide range of languages ‣ web frontend ‣ good & complete documentation ‣ professional as well as community support 20 An overview of other
  37. 37. Thank you ‣ Further questions? ‣ Follow me on twitter/github: @mchacki ‣ Write me a mail: mchacki@arangodb.com ‣ Join or google group: https://groups.google.com/forum/#!forum/arangodb ‣ Visit us at our booth ‣ Free T-shirts 21

×