Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Event sourcing - what could go wrong?

268 views

Published on

Slides from 4Developers Wrocław 2018

Published in: Software
  • Be the first to comment

Event sourcing - what could go wrong?

  1. 1. Event Sourcing - what could go wrong? Andrzej Ludwikowski
  2. 2. About me ➔ ➔ aludwikowski.blogspot.com ➔ github.com/aludwiko ➔ @aludwikowski
  3. 3. What is Event Sourcing? DB Order { items=[itemA, itemB] }
  4. 4. What is Event Sourcing? DB DB Order { items=[itemA, itemB] } ItemAdded(itemA) ItemAdded(itemC) ItemRemoved(itemC) ItemAdded(itemB)
  5. 5. History
  6. 6. History ● 9000 BC, Mesopotamian Clay Tablets, e.g. for market transactions
  7. 7. History ● 2005, Event Sourcing “Enterprise applications that use Event Sourcing are rarer, but I have seen a few applications (or parts of applications) that use it.”
  8. 8. Why Event Sourcing? ● complete log of every state change ● debugging ● performance ● scalability
  9. 9. ES and CQRS Command Service Domain Events Client Query Service Data access Commands Queries Read modelRead modelRead models
  10. 10. ES and CQRS level 1 Command Service Domain Events Client Query Service Data access Commands Queries Read modelRead modelRead models Transaction
  11. 11. ES and CQRS level 1 ● Entry-level, synchronous & transactional event sourcing ● slick-eventsourcing
  12. 12. ES and CQRS level 1 + easy to implement + easy to reason about + 0 eventual consistency - performance - scalability
  13. 13. ES and CQRS level 2 Command Service Domain Events Client Query Service Data access Commands Queries Read modelRead modelRead models Updater Transaction
  14. 14. ES and CQRS level 2 +/- performance +/- scalability - eventual consistency - increased events DB load - lags
  15. 15. ES and CQRS level 3 Command Service Domain Events Client Query Service Data access Commands Queries Read modelRead modelRead models Updater Transaction event bus
  16. 16. ES and CQRS level 3.1 Command Service Domain Events Client Query Service Data access Commands Queries Read modelRead modelRead models Updater event bus Transaction
  17. 17. ES and CQRS level 3.1.1 Command Service Domain Events Client Query Service Data access Commands Queries Read modelRead modelRead models Updater event bus Transaction
  18. 18. ES and CQRS level 3.1.1 Command Service Domain Events Client Query Service Data access Commands Queries Read modelRead modelRead models Updater event bus Transaction At-least-once delivery
  19. 19. ES and CQRS level 3.1.1 Command Service Domain Events Client Query Service Data access Commands Queries Read modelRead modelRead models Updater event bus Transaction
  20. 20. ES and CQRS level 3.1.1 Command Service Domain Events Client Query Service Data access Commands Queries Read modelRead modelRead models Updater event bus Transaction
  21. 21. ES and CQRS level 3.1.1 Command Service Domain Events Client Query Service Data access Commands Queries Read modelRead modelRead models Updater event bus Transaction
  22. 22. ES and CQRS level 3.1.1 Command Service Domain Events Client Query Service Data access Commands Queries Read modelRead modelRead models Updater event bus Transaction
  23. 23. ES and CQRS level 3.1.1 Command Service Domain Events Client Query Service Data access Commands Queries Read modelRead modelRead models Updater event bus Transaction
  24. 24. ES and CQRS level 3.1.1 Command Service Domain Events Client Query Service Data access Commands Queries Read modelRead modelRead models Updater event bus Transaction ?
  25. 25. ES and CQRS level 3.2 Events Client Query Service Data access Commands Queries Read modelRead modelRead models Updater event bus Command Service Domain Command Service Domain Command Service Domain Transaction Sharded Cluster
  26. 26. ES and CQRS level 3.x + performance + scalability - eventual consistency - complex implementation
  27. 27. ES and CQRS alternatives ● Change Capture Data (CDC) logging instead of message queue? ● message queue instead of DB?
  28. 28. ES implementation? ● custom ● library ● framework
  29. 29. ES from domain perspective ● commands, events, state ● 2 main methods on state ○ process(command: Command): List[Event] ○ apply(event: Event): State
  30. 30. ES from application perspective ● snapshotting ● fail-over ● recover ● debugging ● sharding ● serialization & schema evolution ● concurrency access ● etc.
  31. 31. ES packaging ● github.com/aludwiko/event-sourcing-akka-persistence
  32. 32. ES packaging ● snapshotting ● fail-over ● recover ● debugging ● sharding ● serialization & schema evolution ● concurrency access ● etc.
  33. 33. ES packaging ● domain logic ● domain validation ● 0 framework/library imports
  34. 34. library vs. framework ● Akka Persistence vs. Lagom
  35. 35. Event storage ● file ● RDBMS ● Event Store ● MongoDB ● Kafka ● Cassandra
  36. 36. Event storage for Akka Persistence ● file ● RDBMS ● Event Store ● MongoDB ● Kafka ● Cassandra
  37. 37. akka-persistence-jdbc trap
  38. 38. akka-persistence-jdbc trap
  39. 39. Cassandra perfect for ES? ● partitioning by design ● replication by design ● leaderless (no single point of failure) ● optimised for writes (2 nodes = 100 000 tx/s) ● near-linear horizontal scaling
  40. 40. ScyllaDB ? ● Cassandra without JVM ○ same protocol, SSTable compatibility ● C++ and Seastar lib ● 1,000,000 IOPS ● not fully supported by Akka Persistence
  41. 41. Event serialization ● plain text ○ JSON ○ XML ○ YAML ● binary ○ java serialization ○ Avro ○ Protocol Buffers (Protobuf) ○ Thrift ○ Kryo
  42. 42. Plain text Binary human-readable deserialization required
  43. 43. Plain text Binary human-readable deserialization required precision issues (JSON IEEE 754, DoS) -
  44. 44. Plain text Binary human-readable deserialization required precision issues (JSON IEEE 754, DoS) - storage consumption compress
  45. 45. Plain text Binary human-readable deserialization required precision issues (JSON IEEE 754, DoS) - storage consumption compress slow fast
  46. 46. Plain text Binary human-readable deserialization required precision issues (JSON IEEE 754, DoS) - storage consumption compress slow fast poor schema evolution support full schema evolution support
  47. 47. Binary ● java serialization ● Avro ● Protocol Buffers (Protobuf) ● Thrift ● Kryo
  48. 48. Binary ● java serialization ● Avro ● Protocol Buffers (Protobuf) ● Thrift ● Kryo
  49. 49. Binary ● java serialization ● Avro ● Protocol Buffers (Protobuf) ● Thrift ● Kryo
  50. 50. Binary ● java serialization ● Avro ● Protocol Buffers (Protobuf) ● Thrift ● Kryo
  51. 51. Multi-language support ● Avro ○ C, C++, C#, Go, Haskell, Java, Perl, PHP, Python, Ruby, Scala ● Protocol Buffers (Protobuf) ○ even more than Avro
  52. 52. Speed https://code.google.com/archive/p/thrift-protobuf-compare/wikis/Benchmarking.wiki
  53. 53. Size https://code.google.com/archive/p/thrift-protobuf-compare/wikis/Benchmarking.wiki
  54. 54. Full compatibility Application Events ● backward - V2 can read V1 V1 V2
  55. 55. ● forward - V2 can read V3 Full compatibility Application Events V1, V2 V2 Application Application V2 V3
  56. 56. Schema evolution - full compatibility Protocol Buffers Avro Add field + (optional) + (default value) Remove field + + (default value) Rename field + + (aliases) https://martin.kleppmann.com/2012/12/05/schema-evolution-in-avro-protocol-buffers-thrift.html
  57. 57. Protobuf schema management
  58. 58. Protobuf schema management
  59. 59. Protobuf schema management ● def toDomain(event: UserCreatedEvent): UserEvent.UserCreated ● def toSerializable(event: UserEvent.UserCreated): UserCreatedEvent
  60. 60. Protobuf schema management + clean domain - a lot of boilerplate code
  61. 61. Avro schema management
  62. 62. Avro deserialization Bytes Deserialization Object Reader SchemaWriter Schema
  63. 63. Avro writer schema source ● add schema to the payload ● custom solution ○ schema in /resources ○ schema in external storage (must be fault-tolerant) ○ Darwin project ● Schema Registry
  64. 64. Avro schema management
  65. 65. Protocol Buffers vs. Avro
  66. 66. Avro schema management + less boilerplate code +/- clean domain - reader & writer schema distribution
  67. 67. Memory consumption
  68. 68. Immutable vs. mutable state? ● add/remove ImmutableList 17.496 ops/s ● add/remove TreeMap 2201.731 ops/s
  69. 69. Fixing state ● healing command
  70. 70. Updating all aggregates User(id)Command(user_id) Event(user_id)Event(user_id)Event(user_id)
  71. 71. Event + seq_noEvent + seq_no Handling duplicates Events Read modelRead modelRead models Updater event bus Event + seq_no At-least-once delivery
  72. 72. Broken read model Events ad model ead model Read models Updater event bus
  73. 73. Broken read model Events ad model ead model Read models Updater event bus read model update + offset (manual offset management)
  74. 74. Multi aggregate update ● rethink aggregates boundaries ● compensating action ○ optimistic ○ pessimistic
  75. 75. Pessimistic compensation action User account Cinema hall
  76. 76. Pessimistic compensation action User account Cinema hall charged
  77. 77. Pessimistic compensation action User account Cinema hall charged booked
  78. 78. Pessimistic compensation action User account Cinema hall charged booked sold out
  79. 79. Pessimistic compensation action User account Cinema hall charged booked booked sold out
  80. 80. Pessimistic compensation action User account Cinema hall charged booked booked sold out
  81. 81. Pessimistic compensation action User account Cinema hall charged booked booked sold out refund
  82. 82. Optimistic compensation action User account Cinema hall charged booked sold out
  83. 83. Optimistic compensation action User account Cinema hall booked booked sold out overbooked
  84. 84. Optimistic compensation action User account Cinema hall charged booked booked sold out overbooked ?
  85. 85. Saga, post-processing Command Service Domain Events Client Query Service Data access Commands Queries Read modelRead modelRead models Updater event bus Transaction
  86. 86. Saga, post-processing Command Service Domain Events Query Service Data access Commands Queries Read modelRead modelRead models Updater event bus Transaction
  87. 87. Saga, post-processing Command Service Domain Events Client Query Service Data access Commands Queries Read modelRead modelRead models Updater event bus Transaction
  88. 88. Saga, post-processing ● Sagas with ES ● DDD, Saga & Event-sourcing
  89. 89. ES with RODO/GDPR ● “right to forget” with: ○ data shredding (and/or deleting) ■ events, state, views, read models ○ retention policy ■ message brokers, backups, logs ○ data before RODO migration
  90. 90. ES and CQRS level 3.2 Events Client Query Service Data access Commands Queries Read modelRead modelRead models Updater event bus Command Service Domain Command Service Domain Command Service Domain Transaction Sharding Clustering
  91. 91. Cluster = split brain 1 5 4 3 Load balancer 2
  92. 92. Cluster = split brain 1 5 4 3 Load balancer 2 User(1) Command(1)
  93. 93. Cluster = split brain 1 5 4 3 Load balancer 2 User(1)
  94. 94. Cluster = split brain 1 5 4 3 Load balancer 2 User(1) Command(1) User(1)
  95. 95. Cluster = split brain 1 5 4 3 Load balancer 2 User(1) Command(1) User(1) Command(1)
  96. 96. Cluster best practises ● very good monitoring & alerting ● a lot of failover tests ● cluster also on dev/staging ● keep it as small as possible (code base, number of nodes, etc.)
  97. 97. Summary ● understand event/state schema evolution ● eventual consistency is your friend ● scaling is complex ● log-based processing mindset
  98. 98. WE WANT YOU
  99. 99. About me ➔ ➔ aludwikowski.blogspot.com ➔ github.com/aludwiko ➔ @aludwikowski

×