Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Data Architecture –
Not Just for
Microservices
Eberhard Wolff
@ewolff
Fellow
http://continuous-delivery-buch.de/ http://continuous-delivery-book.com/
http://microservices-buch.de/ http://microservices-book.com/
http://microservices-book.com/
primer.html
http://microservices-buch.de/
ueberblick.html
FREE!!!!
Classic Data Architecture
> Centralized databases
> …or services that provide data
> Ensures consistency across systems
> ...
Classic Data Architecture
Billing
Order
Process
CRM
Order
Who is using a
centralized database?
Who likes the
centralized database?
Microservices:
Definition
> No consistent definition
> Microservices are modules
> Independent deployment units
> E.g. pro...
Microservices: Definition
Server /
Container
Server /
Container
Micro
Service
Micro
Service
Why Microservices?
> Develop a feature
> …bring it into production
> ...with no coordination
> Independent scaling
> Free ...
Microservices aim for
decoupling
Microservices & Data
Server /
Container
Server /
Container
Micro
Service
Micro
Service
Order Schema
Microservices & Data
Server /
Container
Server /
Container
Micro
Service
Micro
Service
Order Schema
Microservices & Data
> Decoupling for data, too
> Separate data storage
Data Microservices
Server /
Container
Server /
Container
Micro
Service
Micro
Service
Order Data
Microservice
Data Microservices
Server /
Container
Micro
Service
Order Data
Microservice
Customer Data
Microservice
Remote calls
influe...
Data Microservice
> Change two microservices if new feature
requires change to data schema
> Transactions?
> But: data in ...
Data microservice
limits decoupling.
Encapsulation
> Information hiding
> Hide the internal data structure
> Provide access only through a well defined
interfa...
Violates Encapsulation
Billing
Order
Process
CRM
Order
Shared
data model
Logic
Violates Encapsulation
Server /
Container
Server /
Container
Micro
Service
Micro
Service
Order Data
Microservice Shared
da...
Separate Databases
Server /
Container
Server /
Container
Micro
Service
Micro
Service
Order Order
Different Databases
Server /
Container
Server /
Container
Micro
Service
Micro
Service
neo4j Oracle
Different Databases
> “Polyglot persistence”
> Use the best tool for the job
> Technology freedom
– advantage of microserv...
Separate Schema
Server /
Container
Server /
Container
Micro
Service
Micro
Service
Oracle
Schema Schema
Separate Schemas
> Less effort
> Decoupled data models
> ...but limited independent scaling and
robustness
Billing
Order
Process
CRM
OrderOrder Order
Redundancy!!!
Domain-driven Design
Domain-driven Design
> 2004
> Still very relevant
> By Eric Evans
> Focus on part IV
> Free reference:
http://domainlangua...
Domain-driven Design
> Domain-driven Design
Destilled
> Vaughn Vernon
> Compact
> Book focuses on content
of next slides
>...
Order
Shipping address
Tracking #
Items
Item Categories
Priority shipping
Customs #
Account #
...
Credit card #
Order #
My Domain Model
is a mess!
Bounded Context
> Domain model is only valid for one context
> There is no universal data model!
> See all failed SOA atte...
Order
Shipping address
Tracking #
Items
Item Categories
Priority shipping
Customs #
Account #
...
Credit card #
Order #
Cu...
Billing
Order
Process
CRM
OrderOrder Order
Bounded Context
> Microservice =
BOUNDED CONTEXTS
> Changes for new features are local
> …even if data models need to be c...
Billing
Order
Process
CRM
OrderOrder Order
Redundancy?
Redundancy?
Not really
Bounded Context
What about
basic data of an
order?
Strategic Design
> How do BOUNDED CONTEXTS relate to each
other?
> Context can have relationships
> DDD defines several re...
Shared Kernel
> Subset of a model
> …that two teams share
> Eric Evans: Including code and database
> Microservices: Just ...
Anti-corruption Layer
> Don’t let e.g. a legacy model influence a
new model
> Isolate model by additional layer
> No need ...
Context Relationships
> Team = Deployment Unit = BOUNDED CONTEXT
> Context Relationships define how BOUNDED
CONTEXT are us...
Coordination
Effort
Shared BOUNDED CONTEXT
SHARED KERNEL
CUSTOMER / SUPPLIER
ANTICORRUPTION LAYER
CONFORMIST
SEPARATE WAYS...
Context Map
Context Map
> Show the different BOUNDED CONTEXT
> …and the relation to each other
> BOUNDED CONTEXT might be microservice...
Order ProcessRegistration
Delivery Billing
Anticorruption
Layer
CustomerSupplier
Customer
Supplier
Billing
Order
Process
CRM
Shared
Kernel
Order
Additional
data
Additional
data
Additional
data
Order
Data
Centralized
Shared Kernel
> Ensures consistency
> ...but needs to be called for a lot of
operations
> Resilience / perform...
Billing
Order
Process
CRM
Shared
Kernel
Order
Shared
Kernel
Order
Shared
Kernel
Order
Additional
data
Additional
data
Addi...
Decentralized
Shared Kernel
> Might be inconsistent
> ...but all data for all requests is available in
the local database
...
How to Replicate
Data?
Database Replication
> Built into the database
> Replicate schema across database instances
> But: Microservices have sepa...
Replication with
Events
Events
> Later addition to Domain-driven Design
> Events with a business meaning
> Decouple time:
Asynchronous
> Decouple ...
Billing
Order
Process
CRM
Shared
Kernel
Order
Shared
Kernel
Order
Shared
Kernel
Order
Additional
data
Additional
data
Addi...
Events & Data Replication
> Events lead to data replication
> i.e. each system stores information it
received in an event
...
Batch Replication
Batch
> Get all data
> Provide API
> …to decouple schema
> Copy interesting data into local database
Billing
Order
Process
CRM
Shared
Kernel
Order
Shared
Kernel
Order
Shared
Kernel
Order
Additional
data
Additional
data
Addi...
Batch & Data Replication
> Easy to repair inconsistencies
> Batch run at specific points
> i.e. updates take time
> Data n...
CAP: Challenge for
Replication
CAP Theorem
> Consistency
> All nodes see the same data
> Availability
> Node failures do not prevent survivors
from opera...
CAP Theorem: P
> Network partitions do occur
> Even with highly available network
hardware
> Also: very slow response = pa...
CAP Theorem: C or A?
> Node cannot access other nodes
> Might have missed updates
> A, not C:
Answer with a potentially wr...
Billing
Order
Process
CRM
Shared
Kernel
Order
Shared
Kernel
Order
Shared
Kernel
Order
Additional
data
Additional
data
Addi...
Conclusion
Classic:
Centralized
Database
Microservices:
private
database
decoupling
Data Microservices:
Consistent but
resilience / p...
Redundant Data or
Bounded Context?
Batch
Database
Replication
Events
Redundancy?
Context Map and
Context Relations
Replica...
Decentralize data!
Data Architecture not Just for Microservices
Data Architecture not Just for Microservices
Data Architecture not Just for Microservices
Upcoming SlideShare
Loading in …5
×

Data Architecture not Just for Microservices

3,663 views

Published on

Discusses data architecture and how microservices might influence it. Held at WJAX 2016.

Published in: Software

Data Architecture not Just for Microservices

  1. 1. Data Architecture – Not Just for Microservices Eberhard Wolff @ewolff Fellow
  2. 2. http://continuous-delivery-buch.de/ http://continuous-delivery-book.com/
  3. 3. http://microservices-buch.de/ http://microservices-book.com/
  4. 4. http://microservices-book.com/ primer.html http://microservices-buch.de/ ueberblick.html FREE!!!!
  5. 5. Classic Data Architecture > Centralized databases > …or services that provide data > Ensures consistency across systems > …for data model > ...and updates to data > Reuse
  6. 6. Classic Data Architecture Billing Order Process CRM Order
  7. 7. Who is using a centralized database?
  8. 8. Who likes the centralized database?
  9. 9. Microservices: Definition > No consistent definition > Microservices are modules > Independent deployment units > E.g. processes, Docker container > Microservice owned by one team
  10. 10. Microservices: Definition Server / Container Server / Container Micro Service Micro Service
  11. 11. Why Microservices? > Develop a feature > …bring it into production > ...with no coordination > Independent scaling > Free choice of technology > Robustness > Security
  12. 12. Microservices aim for decoupling
  13. 13. Microservices & Data Server / Container Server / Container Micro Service Micro Service Order Schema
  14. 14. Microservices & Data Server / Container Server / Container Micro Service Micro Service Order Schema
  15. 15. Microservices & Data > Decoupling for data, too > Separate data storage
  16. 16. Data Microservices Server / Container Server / Container Micro Service Micro Service Order Data Microservice
  17. 17. Data Microservices Server / Container Micro Service Order Data Microservice Customer Data Microservice Remote calls influence performance availability No transaction across customer and order
  18. 18. Data Microservice > Change two microservices if new feature requires change to data schema > Transactions? > But: data in one place > No consistency issues
  19. 19. Data microservice limits decoupling.
  20. 20. Encapsulation > Information hiding > Hide the internal data structure > Provide access only through a well defined interface > Data and databases should not be exported
  21. 21. Violates Encapsulation Billing Order Process CRM Order Shared data model Logic
  22. 22. Violates Encapsulation Server / Container Server / Container Micro Service Micro Service Order Data Microservice Shared data model Logic
  23. 23. Separate Databases Server / Container Server / Container Micro Service Micro Service Order Order
  24. 24. Different Databases Server / Container Server / Container Micro Service Micro Service neo4j Oracle
  25. 25. Different Databases > “Polyglot persistence” > Use the best tool for the job > Technology freedom – advantage of microservices > …but extra effort > Backup, disaster recovery etc. > Not as easy as e.g. different frameworks
  26. 26. Separate Schema Server / Container Server / Container Micro Service Micro Service Oracle Schema Schema
  27. 27. Separate Schemas > Less effort > Decoupled data models > ...but limited independent scaling and robustness
  28. 28. Billing Order Process CRM OrderOrder Order
  29. 29. Redundancy!!!
  30. 30. Domain-driven Design
  31. 31. Domain-driven Design > 2004 > Still very relevant > By Eric Evans > Focus on part IV > Free reference: http://domainlanguage.com/ ddd/reference/
  32. 32. Domain-driven Design > Domain-driven Design Destilled > Vaughn Vernon > Compact > Book focuses on content of next slides > Soon: German translation
  33. 33. Order Shipping address Tracking # Items Item Categories Priority shipping Customs # Account # ... Credit card # Order #
  34. 34. My Domain Model is a mess!
  35. 35. Bounded Context > Domain model is only valid for one context > There is no universal data model! > See all failed SOA attempts
  36. 36. Order Shipping address Tracking # Items Item Categories Priority shipping Customs # Account # ... Credit card # Order # Customs Order Recommen- dations Order Tracking Order Shipping address Tracking # Item Categories Priority shipping Customs # Payment Order Account # Credit card #
  37. 37. Billing Order Process CRM OrderOrder Order
  38. 38. Bounded Context > Microservice = BOUNDED CONTEXTS > Changes for new features are local > …even if data models need to be changed
  39. 39. Billing Order Process CRM OrderOrder Order
  40. 40. Redundancy?
  41. 41. Redundancy? Not really
  42. 42. Bounded Context
  43. 43. What about basic data of an order?
  44. 44. Strategic Design > How do BOUNDED CONTEXTS relate to each other? > Context can have relationships > DDD defines several relationship patterns
  45. 45. Shared Kernel > Subset of a model > …that two teams share > Eric Evans: Including code and database > Microservices: Just sharing a model
  46. 46. Anti-corruption Layer > Don’t let e.g. a legacy model influence a new model > Isolate model by additional layer > No need to modify the old system
  47. 47. Context Relationships > Team = Deployment Unit = BOUNDED CONTEXT > Context Relationships define how BOUNDED CONTEXT are used… > ...and how much teams need to collaborate
  48. 48. Coordination Effort Shared BOUNDED CONTEXT SHARED KERNEL CUSTOMER / SUPPLIER ANTICORRUPTION LAYER CONFORMIST SEPARATE WAYS Limit for Microservices
  49. 49. Context Map
  50. 50. Context Map > Show the different BOUNDED CONTEXT > …and the relation to each other > BOUNDED CONTEXT might be microservices > ...or communication links
  51. 51. Order ProcessRegistration Delivery Billing Anticorruption Layer CustomerSupplier Customer Supplier
  52. 52. Billing Order Process CRM Shared Kernel Order Additional data Additional data Additional data Order Data
  53. 53. Centralized Shared Kernel > Ensures consistency > ...but needs to be called for a lot of operations > Resilience / performance / transactions > Have one master as the source of truth
  54. 54. Billing Order Process CRM Shared Kernel Order Shared Kernel Order Shared Kernel Order Additional data Additional data Additional data
  55. 55. Decentralized Shared Kernel > Might be inconsistent > ...but all data for all requests is available in the local database > Better resilience… > ...and performance
  56. 56. How to Replicate Data?
  57. 57. Database Replication > Built into the database > Replicate schema across database instances > But: Microservices have separated schemas > Every Microservice might have different data > …so database replication is not a good fit
  58. 58. Replication with Events
  59. 59. Events > Later addition to Domain-driven Design > Events with a business meaning > Decouple time: Asynchronous > Decouple logic: System can handle event as it pleases
  60. 60. Billing Order Process CRM Shared Kernel Order Shared Kernel Order Shared Kernel Order Additional data Additional data Additional data New Order Event
  61. 61. Events & Data Replication > Events lead to data replication > i.e. each system stores information it received in an event > Data stored in separate schema > Very decoupled > Hard to repair inconsistencies
  62. 62. Batch Replication
  63. 63. Batch > Get all data > Provide API > …to decouple schema > Copy interesting data into local database
  64. 64. Billing Order Process CRM Shared Kernel Order Shared Kernel Order Shared Kernel Order Additional data Additional data Additional data BatchBatch API API
  65. 65. Batch & Data Replication > Easy to repair inconsistencies > Batch run at specific points > i.e. updates take time > Data not consistent across microservices
  66. 66. CAP: Challenge for Replication
  67. 67. CAP Theorem > Consistency > All nodes see the same data > Availability > Node failures do not prevent survivors from operating > Partition Tolerance > System continues to operate despite arbitrary message loss C P A
  68. 68. CAP Theorem: P > Network partitions do occur > Even with highly available network hardware > Also: very slow response = partition > Need to deal with P
  69. 69. CAP Theorem: C or A? > Node cannot access other nodes > Might have missed updates > A, not C: Answer with a potentially wrong answer > C, not A: Don’t answer – the answer might be wrong
  70. 70. Billing Order Process CRM Shared Kernel Order Shared Kernel Order Shared Kernel Order Additional data Additional data Additional data New Order Event inconsistent or unavailable
  71. 71. Conclusion
  72. 72. Classic: Centralized Database Microservices: private database decoupling Data Microservices: Consistent but resilience / performance / transactions / decoupling? Database per Microservice: Polyglot Persistence Schema per Microservice: Simple infrastructure
  73. 73. Redundant Data or Bounded Context? Batch Database Replication Events Redundancy? Context Map and Context Relations Replication CAP e.g. Shared Kernel
  74. 74. Decentralize data!

×