Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Event sourcing - what could possible go wrong?

44 views

Published on

from Wrocław JUG 08.10.2019

Published in: Engineering
  • Be the first to comment

  • Be the first to like this

Event sourcing - what could possible go wrong?

  1. 1. Event Sourcing - what could go wrong? Andrzej Ludwikowski
  2. 2. About me ➔ ➔ ludwikowski.info ➔ github.com/aludwiko ➔ @aludwikowski
  3. 3. SoftwareMill
  4. 4. SoftwareMill
  5. 5. SoftwareMill
  6. 6. SoftwareMill JVM BLOGGERS
  7. 7. SoftwareMill ● Hibernate Envers ● Alpakka Kafka ● Sttp ● Scala Clippy ● Quicklens ● MacWire
  8. 8. What is Event Sourcing? DB Order { items=[itemA, itemB] }
  9. 9. What is Event Sourcing? DB DB Order { items=[itemA, itemB] } ItemAdded(itemA) ItemAdded(itemC) ItemRemoved(itemC) ItemAdded(itemB)
  10. 10. History ● 9000 BC, Mesopotamian Clay Tablets, e.g. for market transactions
  11. 11. History ● 2005, Event Sourcing “Enterprise applications that use Event Sourcing are rarer, but I have seen a few applications (or parts of applications) that use it.”
  12. 12. Why Event Sourcing? ● complete log of every state change ● debugging ● performance ● scalability
  13. 13. ES and CQRS Command Service Domain Events Client Query Service Data access Commands Queries Read modelRead modelRead models
  14. 14. ES and CQRS level 1 Command Service Domain Events Client Query Service Data access Commands Queries Read modelRead modelRead models Transaction
  15. 15. ES and CQRS level 1 ● Entry-level, synchronous & transactional event sourcing ● slick-eventsourcing
  16. 16. ES and CQRS level 1 + easy to implement + easy to reason about + 0 eventual consistency - performance - scalability
  17. 17. ES and CQRS level 2 Command Service Domain Events Client Query Service Data access Commands Queries Read modelRead modelRead models Updater Transaction
  18. 18. ES and CQRS level 2 +/- performance +/- scalability - eventual consistency - increased events DB load ? - lags
  19. 19. ES and CQRS level 3 Command Service Domain Events Client Query Service Data access Commands Queries Read modelRead modelRead models Updater Transaction event bus
  20. 20. ES and CQRS level 3.1 Command Service Domain Events Client Query Service Data access Commands Queries Read modelRead modelRead models Updater event bus Transaction
  21. 21. ES and CQRS level 3.1.1 Command Service Domain Events Client Query Service Data access Commands Queries Read modelRead modelRead models Updater event bus Transaction
  22. 22. ES and CQRS level 3.1.1 Command Service Domain Events Client Query Service Data access Commands Queries Read modelRead modelRead models Updater event bus Transaction At-least-once delivery
  23. 23. ES and CQRS level 3.1.1 Command Service Domain Events Client Query Service Data access Commands Queries Read modelRead modelRead models Updater event bus Transaction
  24. 24. ES and CQRS level 3.1.1 Command Service Domain Events Client Query Service Data access Commands Queries Read modelRead modelRead models Updater event bus Transaction
  25. 25. ES and CQRS level 3.1.1 Command Service Domain Events Client Query Service Data access Commands Queries Read modelRead modelRead models Updater event bus Transaction
  26. 26. ES and CQRS level 3.1.1 Command Service Domain Events Client Query Service Data access Commands Queries Read modelRead modelRead models Updater event bus Transaction
  27. 27. ES and CQRS level 3.1.1 Command Service Domain Events Client Query Service Data access Commands Queries Read modelRead modelRead models Updater event bus Transaction
  28. 28. ES and CQRS level 3.1.1 Command Service Domain Events Client Query Service Data access Commands Queries Read modelRead modelRead models Updater event bus Transaction ?
  29. 29. ES and CQRS level 3.2 Events Client Query Service Data access Commands Queries Read modelRead modelRead models Updater event bus Command Service Domain Command Service Domain Command Service Domain Transaction ` Sharded Cluster
  30. 30. ES and CQRS level 3.x + performance + scalability - eventual consistency - complex implementation
  31. 31. ES and CQRS alternatives ● Change Capture Data (CDC) logging instead of event bus? ● event bus instead of DB?
  32. 32. Not covered but worth to check ● Command Sourcing ● Event Collaboration
  33. 33. ES implementation? ● custom ● library ● framework
  34. 34. ES from domain perspective ● commands, events, state ● 2 methods on state ○ process(command: Command): List[Event] ○ apply(event: Event): State
  35. 35. ES from application perspective ● snapshotting ● fail-over ● recover ● debugging ● sharding ● serialization & schema evolution ● concurrency access ● etc.
  36. 36. import javax.persistence.*; import java.util.List; @Entity public class Issue { @EmbeddedId private IssueId id; private String name; private IssueStatus status; @OneToMany(cascade = CascadeType.MERGE) private List<IssueComment> comments; ... public void changeStatusTo(IssueStatus newStatus) { if (this.status == IssueStatus.DONE && newStatus == IssueStatus.NEW || this.status == IssueStatus.NEW && newStatus == IssueStatus.DONE) { throw new RuntimeException(String.format("Cannot change issue status from %s to %s", this.status, newStatus)); } this.status = newStatus; } ... }
  37. 37. import org.axonframework.commandhandling.* import org.axonframework.eventsourcing.* @Aggregate(repository = "userAggregateRepository") public class User { @AggregateIdentifier private UserId userId; private String passwordHash; @CommandHandler public boolean handle(AuthenticateUserCommand cmd) { boolean success = this.passwordHash.equals(hashOf(cmd.getPassword())); if (success) { apply(new UserAuthenticatedEvent(userId)); } return success; } @EventSourcingHandler public void on(UserCreatedEvent event) { this.userId = event.getUserId(); this.passwordHash = event.getPassword(); } private String hashOf(char[] password) { return DigestUtils.sha1(String.valueOf(password)); } }
  38. 38. import akka.Done import com.lightbend.lagom.scaladsl.* import play.api.libs.json.{Format, Json} import com.example.auction.utils.JsonFormats._ class UserEntity extends PersistentEntity { override def initialState = None override def behavior: Behavior = { case Some(user) => Actions().onReadOnlyCommand[GetUser.type, Option[User]] { case (GetUser, ctx, state) => ctx.reply(state) }.onReadOnlyCommand[CreateUser, Done] { case (CreateUser(name), ctx, state) => ctx.invalidCommand("User already exists") } case None => Actions().onReadOnlyCommand[GetUser.type, Option[User]] { case (GetUser, ctx, state) => ctx.reply(state) }.onCommand[CreateUser, Done] { case (CreateUser(name), ctx, state) => ctx.thenPersist(UserCreated(name))(_ => ctx.reply(Done)) }.onEvent { case (UserCreated(name), state) => Some(User(name)) } } }
  39. 39. ES packaging ● github.com/aludwiko/event-sourcing-akka-persistence
  40. 40. import java.time.Instant import info.ludwikowski.es.user.domain.UserCommand.* import info.ludwikowski.es.user.domain.UserEvent.* import scala.util.{Failure, Success, Try} final case class User (userId: UserId, name: String, email: Email) { def applyEvent(userEvent: UserEvent): Try[User] = ??? //pattern matching def process(userCommand: UserCommand): Try[List[UserEvent]] = ??? //pattern matching } object User { def from(u: UserCreated): User = User(u.userId, u.name, u.email) }
  41. 41. ES packaging ● snapshotting ● fail-over ● recover ● debugging ● sharding ● serialization & schema evolution ● concurrency access ● etc.
  42. 42. ES packaging ● domain logic ● domain validation ● 0 ES framework imports
  43. 43. library vs. framework ● Akka Persistence vs. Lagom
  44. 44. Event storage ● file ● RDBMS ● Event Store ● MongoDB ● Kafka ● Cassandra
  45. 45. Event storage for Akka Persistence ● file ● RDBMS ● Event Store ● MongoDB ● Kafka ● Cassandra
  46. 46. akka-persistence-jdbc trap val theTag = s"%$tag%" sql""" SELECT "#$ordering", "#$deleted", "#$persistenceId", "#$sequenceNumber", "#$message", "#$tags" FROM ( SELECT * FROM #$theTableName WHERE "#$tags" LIKE $theTag AND "#$ordering" > $theOffset AND "#$ordering" <= $maxOffset ORDER BY "#$ordering" ) WHERE rownum <= $max"""
  47. 47. akka-persistence-jdbc trap SELECT * FROM events_journal WHERE tags LIKE ‘%some_tag%’;
  48. 48. Cassandra perfect for ES? ● partitioning by design ● replication by design ● leaderless (no single point of failure) ● optimised for writes (2 nodes = 100 000 tx/s) ● near-linear horizontal scaling
  49. 49. ScyllaDB ? ● Cassandra without JVM ○ same protocol, SSTable compatibility ● C++ and Seastar lib ● up to 1,000,000 IOPS/node ● not fully supported by Akka Persistence
  50. 50. Event serialization ● plain text ○ JSON ○ XML ○ YAML ● binary ○ java serialization ○ Avro ○ Protocol Buffers (Protobuf) ○ Thrift ○ Kryo
  51. 51. Plain text Binary human-readable deserialization required
  52. 52. Plain text Binary human-readable deserialization required precision issues (JSON IEEE 754, DoS) -
  53. 53. Plain text Binary human-readable deserialization required precision issues (JSON IEEE 754, DoS) - storage consumption compress
  54. 54. Plain text Binary human-readable deserialization required precision issues (JSON IEEE 754, DoS) - storage consumption compress slow fast
  55. 55. Plain text Binary human-readable deserialization required precision issues (JSON IEEE 754, DoS) - storage consumption compress slow fast poor schema evolution support full schema evolution support
  56. 56. Binary ● java serialization ● Avro ● Protocol Buffers (Protobuf) ● Thrift ● Kryo
  57. 57. Binary ● java serialization ● Avro ● Protocol Buffers (Protobuf) ● Thrift ● Kryo
  58. 58. Binary ● java serialization ● Avro ● Protocol Buffers (Protobuf) ● Thrift ● Kryo
  59. 59. Binary ● java serialization ● Avro ● Protocol Buffers (Protobuf) ● Thrift ● Kryo
  60. 60. Multi-language support ● Avro ○ C, C++, C#, Go, Haskell, Java, Perl, PHP, Python, Ruby, Scala ● Protocol Buffers (Protobuf) ○ even more than Avro
  61. 61. Speed https://code.google.com/archive/p/thrift-protobuf-compare/wikis/Benchmarking.wiki
  62. 62. Size https://code.google.com/archive/p/thrift-protobuf-compare/wikis/Benchmarking.wiki
  63. 63. Full compatibility Application Events ● backward - V2 can read V1 V1 V2
  64. 64. ● forward - V2 can read V3 Full compatibility Application Events V1, V2 V2 Application Application V2 V3
  65. 65. ● forward - V2 can read V3 Full compatibility Events Read modelRead modelRead models Updater V2V3
  66. 66. Schema evolution - full compatibility Protocol Buffers Avro Add field + (optional) + (default value) Remove field + + (default value) Rename field + + (aliases) https://martin.kleppmann.com/2012/12/05/schema-evolution-in-avro-protocol-buffers-thrift.html
  67. 67. Protobuf schema management //user-events.proto message UserCreatedEvent { string user_id = 1; string operation_id = 2; int64 created_at = 3; string name = 4; string email = 5; } package user.application UserCreatedEvent( userId: String, operationId: String, createdAt: Long, name: String, email: String )
  68. 68. Protobuf schema management package user.domain UserCreated( userId: UserId, operationId: OperationId, createdAt: Instant, name: String, email: Email ) extends UserEvent package user.application UserCreatedEvent( userId: String, operationId: String, createdAt: Long, name: String, email: String )
  69. 69. Protobuf schema management ● def toDomain(event: UserCreatedEvent): UserEvent.UserCreated ● def toSerializable(event: UserEvent.UserCreated): UserCreatedEvent
  70. 70. Protobuf schema management + clean domain - a lot of boilerplate code
  71. 71. Avro schema management package user.domain UserCreated( userId: UserId, operationId: OperationId, createdAt: Instant, name: String, email: Email ) extends UserEvent { "type" : "record", "name" : "UserCreated", "namespace" : "info.ludwikowski.es.user.domain", "fields" : [ { "name" : "userId", "type" : "string" }, { "name" : "operationId", "type" : "string" }, { "name" : "createdAt", "type" : "long" }, { "name" : "name", "type" : "string" }, { "name" : "email", "type" : "string" } ] }
  72. 72. Avro deserialization Bytes Deserialization Object Reader SchemaWriter Schema
  73. 73. Avro writer schema source ● add schema to the payload ● custom solution ○ schema in /resources ○ schema in external storage (must be fault-tolerant) ○ Darwin project ● Schema Registry
  74. 74. Avro schema management package user.domain UserCreated( userId: UserId, operationId: OperationId, createdAt: Instant, name: String, email: Email ) extends UserEvent { "type" : "record", "name" : "UserCreated", "namespace" : "info.ludwikowski.es.user.domain", "fields" : [ { "name" : "userId", "type" : "string" }, { "name" : "operationId", "type" : "string" }, { "name" : "createdAt", "type" : "long" }, { "name" : "name", "type" : "string" }, { "name" : "email", "type" : "string" } ] }
  75. 75. Protocol Buffers vs. Avro { "type" : "record", "name" : "UserCreated", "namespace" : "info.ludwikowski.es.user.domain", "fields" : [ { "name" : "userId", "type" : "string" }, { "name" : "operationId", "type" : "string" }, { "name" : "createdAt", "type" : "long" }, { "name" : "name", "type" : "string" }, { "name" : "email", "type" : "string" } ] } message UserCreatedEvent { string user_id = 1; string operation_id = 2; int64 created_at = 3; string name = 4; string email = 5; }
  76. 76. Avro schema management + less boilerplate code +/- clean domain - reader & writer schema distribution
  77. 77. Avro + less boilerplate code +/- clean domain - reader & writer schema distribution Protobuf + clean domain - a lot of boilerplate code
  78. 78. Avro vs. Protocol Buffers ● The best serialization strategy for Event Sourcing
  79. 79. Memory consumption
  80. 80. Immutable vs. mutable state? ● add/remove ImmutableList 17.496 ops/s ● add/remove TreeMap 2201.731 ops/s
  81. 81. Fixing state ● healing command
  82. 82. Updating all aggregates User(id)Command(user_id) Event(user_id)Event(user_id)Event(user_id)
  83. 83. Handling duplicates Events Read modelRead modelRead models Updater event bus At-least-once delivery
  84. 84. https://www.seriouspuzzles.com/unicorns-in-fairy-land-500pc-jigsaw-puzzle-by-eurographics/
  85. 85. Handling duplicates Events Read modelRead modelRead models Updater event bus At-least-once delivery
  86. 86. Handling duplicates Events Read modelRead modelRead models Updater event bus idempotent updates
  87. 87. Event + seq_noEvent + seq_no Handling duplicates Events Read modelRead modelRead models Updater event bus Event + seq_no read model update + seq_no
  88. 88. Broken read model Events ad model ead model Read models Updater event bus
  89. 89. Broken read model Events ad model ead model Read models Updater event bus read model update + offset (manual offset management)
  90. 90. Multi aggregate transactional update ● rethink aggregates boundaries ● compensating action ○ optimistic ○ pessimistic
  91. 91. Pessimistic compensation action User account Cinema hall
  92. 92. Pessimistic compensation action User account Cinema hall charged
  93. 93. Pessimistic compensation action User account Cinema hall charged booked
  94. 94. Pessimistic compensation action User account Cinema hall charged booked sold out
  95. 95. Pessimistic compensation action User account Cinema hall charged booked booked sold out
  96. 96. Pessimistic compensation action User account Cinema hall charged booked booked sold out
  97. 97. Pessimistic compensation action User account Cinema hall charged booked booked sold out refund
  98. 98. Optimistic compensation action User account Cinema hall charged booked sold out
  99. 99. Optimistic compensation action User account Cinema hall booked booked sold out overbooked
  100. 100. Optimistic compensation action User account Cinema hall charged booked booked sold out overbooked ?
  101. 101. Saga Command Service Domain Events Client Query Service Data access Commands Queries Read modelRead modelRead models Updater event bus Transaction
  102. 102. Saga Command Service Domain Events Query Service Data access Commands Queries Read modelRead modelRead models Updater event bus Transaction
  103. 103. Saga Command Service Domain Events Client Query Service Data access Commands Queries Read modelRead modelRead models Updater event bus Transaction
  104. 104. Saga ● should be persistable ● events order should be irrelevant ● time window limitation ● compensating action must be commutative
  105. 105. Saga ● Sagas with ES ● DDD, Saga & Event-sourcing ● Applying Saga Pattern ● Microservice Patterns
  106. 106. ES with RODO/GDPR ● “right to forget” with: ○ data shredding (and/or deleting) ■ events, state, views, read models ○ retention policy ■ message brokers, backups, logs ○ data before RODO migration
  107. 107. ES and CQRS level 3.2 Events Client Query Service Data access Commands Queries Read modelRead modelRead models Updater event bus Command Service Domain Command Service Domain Command Service Domain Transaction Sharding Clustering
  108. 108. Cluster = split brain 1 5 4 3 Load balancer 2
  109. 109. Cluster = split brain 1 5 4 3 Load balancer 2 User(1) Command(1)
  110. 110. Cluster = split brain 1 5 4 3 Load balancer 2
  111. 111. Cluster = split brain 1 5 4 3 Load balancer 2 User(1)
  112. 112. Cluster = split brain 1 5 4 3 Load balancer 2 User(1) Command(1) User(1)
  113. 113. Cluster = split brain 1 5 4 3 Load balancer 2 User(1) Command(1) User(1) Command(1)
  114. 114. Cluster best practises ● remember about the split brain ● very good monitoring & alerting ● a lot of failover tests ● cluster also on dev/staging ● keep it as small as possible (code base, number of nodes, etc.)
  115. 115. Summary ● carefully choose ES lib/framework ● understand event/command/state schema evolution ● eventual consistency is your friend ● scaling is complex ● database inside-out ● log-based processing mindset
  116. 116. Rate me please :)
  117. 117. Thank you and Q&A ➔ ➔ ludwikowski.info ➔ github.com/aludwiko ➔ @aludwikowski
  118. 118. WE WANT YOU
  119. 119. Reactive Event Sourcing Performance tests with Gatling andrzejludwikowski@gmail.com

×