Download as PDF, PPTX
Uploaded bygvenzl
Data Management in a Microservices World
The document discusses data management in a microservices context, emphasizing the concept of polyglot persistence where different applications and parts of applications utilize various data storage technologies to meet diverse needs. It highlights the evolution of data management from traditional relational databases to multi-model databases and the importance of considering the unique requirements of applications while adopting suitable data models. Additionally, it covers the challenges and considerations in managing data and the integration of containerization and microservices architecture.
More Related Content
![OOW16 - Advanced Architectures for Oracle E-Business Suite [CON6705]](https://cdn.slidesharecdn.com/ss_thumbnails/con6705-elkephelps-advanced-architectures-for-oracle-e-business-suite-160930182612-thumbnail.jpg?width=640&height=640&fit=bounds)
![OOW16 - Maintenance Strategies for Oracle E-Business Suite [CON6725]](https://cdn.slidesharecdn.com/ss_thumbnails/con6725-elkephelps-maintenance-strategies-for-oracle-e-business-suite-160930182637-thumbnail.jpg?width=640&height=640&fit=bounds)
![OOW16 - Oracle E-Business Suite Integration Best Practices [CON6709]](https://cdn.slidesharecdn.com/ss_thumbnails/con6709-oraclee-businesssuiteintegrationbestpractices-160930182601-thumbnail.jpg?width=640&height=640&fit=bounds)
Similar to Data Management in a Microservices World
Data Management in a Microservices World
- 1.
- 2.
- 3. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. | 3 Gerald Venzl • Oracle Database Product Manager •Previous: – Java Developer – Performance Engineer – Enterprise Architect • Twitter: @GeraldVenzl • Blog: https://geraldonit.com
- 4.
- 5.
- 6.
- 7.
- 8.
- 9.
- 10.
- 11.
- 12.
- 13.
- 14.
- 15.
- 16.
- 17.
- 18.
- 19.
- 20.
- 21. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. | Different applications/ µServices have different needs User Sessions Financial Transactions Shopping Cart Recommend. Engine Product Catalog ReportingAnalytics Activity Logs Processing Heavy Writes √ Heavy Reads √ √ √ √ Fast Read/Write √ Data Consistency √ Data Durability √ Analytic √ √ Graph √ Spatial Geo Distribution √ √ √ Data Relational √ √ √ Key/Value √ √ √ Document/JSON √ √ √ Graph √ √
- 22.
- 23.
- 24.
- 25.
- 26.
- 27.
- 28.
- 29.
- 30.
- 31. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. | Multi-model capabilities • The relational model is a logical model • Databases store data not in rows and columns, but in blocks –Blocks are aligned to block storage, i.e. disks and the way they store data – Data is logically represented as rows and columns • Multi-model databases contain logical models to other data formats
- 32. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. | Polyglot Persistence Market Trends • Single-model architectures are most pervasive for ‘edge’ applications – New business & workload requirements •Business applications naturally converge to multi-model architectures – Today’s ‘edge’ applications are tomorrow’s mainstream business applications – Efficiencies of multi-model architecture override advantages of special-purpose systems over time • There will always be single-model polyglot architectures – Because there are always new ‘edge’ applications
- 33.
- 34. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. | Considerations for Polyglot Persistence Multi-model Polyglot: • Benefits of consolidation and standardization • Standardized administration •Consistent data security policies • Simple integration across multiple data formats • Transactions and data consistency Single-model Polyglot: • Benefits of specialization • Specialized APIs • Specialized data formats • Specialized access methods and indexes
- 35.
- 36.
- 37.
- 38.
- 39.
- 40.
- 41.
- 42.
- 43.
- 44.
- 45.
- 46.
- 47.
- 48. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. | The data reasoning problem • Much harder to understand the structure from other domains – Now you have to know how other applications structure their data •Schema-on-read brings an inherent data cleansing problem – As no constraints are enforced at write no guarantees can be assumed • The more data you have, the more complex it gets – Need to hire Data Scientists where SQL used to be sufficient
- 49.
- 50.
- 51.
- 52.
- 53. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. | Change tracking for all DMLs Analysis over entire history of changes via SQL Risk calculations Risk calculation history Flashback Query (SELECT AS OF) “Retrieve risk calculations as of 90 days ago” SELECT * FROMrisk_calculations AS OF (SYSDATE – 90);
- 54. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. | Change tracking for all DMLs Analysis over entire history of changes via SQL Risk calculations Risk calculation history Flashback Query (SELECT AS OF) “Retrieve risk calculations as of 90 days ago” SELECT * FROMrisk_calculations AS OF (SYSDATE – 90);
- 55. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. | Change tracking for all DMLs Analysis over entire history of changes via SQL Risk calculations Risk calculation history Flashback Version Query (SELECT VERSIONS BETWEEN start AND end) “Retrieve all risk calculations changes from two weeks ago” SELECT *FROM risk_calculations VERSIONS BETWEEN SYSDATE-14 AND SYSDATE-7; Flashback Query (SELECT AS OF) “Retrieve risk calculations as of 90 days ago” SELECT * FROM risk_calculations AS OF (SYSDATE – 90);
- 56. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. | Change tracking for all DMLs Analysis over entire history of changes via SQL Risk calculations Risk calculation history Flashback Version Query (SELECT VERSIONS BETWEEN start AND end) “Retrieve all risk calculations changes from two weeks ago” SELECT *FROM risk_calculations VERSIONS BETWEEN SYSDATE-14 AND SYSDATE-7; Flashback Query (SELECT AS OF) “Retrieve risk calculations as of 90 days ago” SELECT * FROM risk_calculations AS OF (SYSDATE – 90);
- 57. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. | Change tracking for all DMLs Analysis over entire history of changes via SQL Risk calculations Risk calculation history Flashback Version Query “Pull entire change history for factor = ’Major’ ” SELECT * FROMrisk_calculations VERSIONS BETWEEN SYSDATE AND MAXVALUE WHERE factor = 'Major’; UNDO
- 58. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. | Change tracking for all DMLs Analysis over entire history of changes via SQL Risk calculations Risk calculation history Flashback Version Query “Pull entire change history for factor = ’Major’ ” SELECT * FROMrisk_calculations VERSIONS BETWEEN SYSDATE AND MAXVALUE WHERE factor = 'Major’; UNDO
- 59. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. | Change tracking for all DMLs Analysis over entire history of changes via SQL Risk calculations Risk calculation history • Fully secure – Encryption at transport and rest –Fine grain access control – Fully automatic, never loosing a single change.
- 60.
- 61.
- 62.
- 63.
- 64.
- 65.
- 66.
- 67.
- 68.
- 69. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. | Containerized databases Deployment transparency Product catalog Recommend engine Cluster EMEA Financial data Recommend engine • PDB Hot Clone – Faster container instantiation •PDB Relocate – Relocate with no downtime Cluster US Product catalog
- 70. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. | Containerized databases Deployment transparency Product catalog Recommend engine Cluster EMEA Financial data Recommend engine • PDB Hot Clone – Faster container instantiation •PDB Relocate – Relocate with no downtime Cluster US Product catalog Product catalog
- 71. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. | Containerized databases Deployment transparency Product catalog Recommend engine Cluster EMEA Financial data Recommend engine • PDB Hot Clone – Faster container instantiation •PDB Relocate – Relocate with no downtime Cluster US Product catalog Product catalog Product catalog
- 72. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. | Containerized databases Deployment transparency Product catalog Recommend engine Cluster EMEA Financial data Recommend engine • PDB Hot Clone – Faster container instantiation •PDB Relocate – Relocate with no downtime Cluster US Product catalog Product catalog Product catalog
- 73. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. | Containerized databases Deployment transparency Financial data Product catalog Cluster B Product catalog Recommend engine • PDB Hot Clone – Faster container instantiation •PDB Relocate – Relocate with no downtime Cluster A Financial data
- 74. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. | Containerized databases Deployment transparency Product catalog Cluster B Product catalog Recommend engine Financial data • PDB Hot Clone – Faster container instantiation •PDB Relocate – Relocate with no downtime Cluster A Financial data
- 75. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. | Containerized databases Deployment transparency Product catalog Cluster B Product catalog Recommend engine Financial data • PDB Hot Clone – Faster container instantiation •PDB Relocate – Relocate with no downtime Cluster A Financial data
- 76.
- 77.
- 78. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. | Oracle Database Sharding Oracle Database for web-scale applications One giant database partitioned into many small databases (shards) Customers Americas Customers Customers Europe Customers Asia • RAC and Data Guard meet needs of over 99% of applications while preserving application transparency • Some Global-Scale OLTP applications prefer to shard massive databases into a farm of smaller databases –Avoid scalability or availability edge cases of a single large database – Willing to customize data model and applications to enable transactions to be automatically routed to the right shard • Native SQL for shardingtables across up to 1000 Shards - Routing of SQL based on shard key, and cross shard queries - Online addition and reorganization of shards - Linear scalability of data, workload, users with isolation
- 79. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. | Oracle Sharding Automated Distribution Enhanced SQL syntax for Sharding … CREATE SHARDEDTABLE Customers ( CustId VARCHAR2(60) NOT NULL, FirstName VARCHAR2(60), LastName VARCHAR2(60), … PRIMARY KEY(CustId), ) PARTITION BY CONSISTENT HASH (CustId) … • SQL syntax for creating sharded tables • Not proprietary APIs as with NoSQL • Creation of a sharded table automatically partitions data across shards • Transparent resharding as data grows • Choice of sharding methods: • System managed - consistent hash • User defined - range, list • Composite - range-hash, list-hash • Common reference data (e.g. Price List) is automatically duplicated on all shards • Supports shard placement in specific geographies to satisfy government data privacy … …
- 80.
- 81.
- 82.
- 83.
- 84.
- 85.
- 86.
- 87.
- 88.
- 89.
- 90.
- 91. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. | Communication Financial data API Shopping cart API JSON Async ? • Was the payment processed or not? • Was the data persisted or not? •If not, how do we replay the payment? • If yes, how do we guarantee that we don’t bill the customer twice? • … Reliable communication without tight coupling
- 92.
- 93. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. | Communication Financial data API Shopping cart API JSON Sync • Possible solution, but… •Services will be tightly coupled • Outage of one system will impact the other system • Prevents service independent scaling Reliable communication without tight coupling
- 94.
- 95.
- 96.
- 97.
- 98.
- 99. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. | Financial data API Oracle Advanced Queueing Shopping cart API JSON Async AQ Reliable communication without tight coupling • Allows reliable communication without coupling • Further increases scalability –REST payloads are never lost and put into one or multiple queues to work off from • Provides transparency – Works for any data – Works for PDBs and non-CDBs AQ
- 100.
- 101.
- 102.
- 103.
- 104.
- 105.
- 106.
- 107.
- 108.
- 109.
- 110.
- 111.