Event Sourcing - what could possibly go wrong?
•
0 likes•317 views
Yet another presentation about Event Sourcing? Yes and no. Event Sourcing is a really great concept. Some could say it’s a Holy Grail of the software architecture. I might agree with that, while remembering that everything comes with a price. This session is a summary of my experience with ES gathered while working on 3 different commercial products. Instead of theoretical aspects, I will focus on possible challenges with ES implementation. What could explode (very often with delayed ignition)? How and where to store events effectively? What are possible schema evolution solutions? How to achieve the highest level of scalability and live with eventual consistency? And many other interesting topics that you might face when experimenting with ES.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Event Sourcing - what could possibly go wrong?
Similar to Event Sourcing - what could possibly go wrong? (20)
More from Andrzej Ludwikowski
Recently uploaded
Recently uploaded (20)
Event Sourcing - what could possibly go wrong?
- 1. Event Sourcing - what could go wrong? Andrzej Ludwikowski
- 2. About me ➔ ➔ ludwikowski.info ➔ github.com/aludwiko ➔ @aludwikowski
- 3. SoftwareMill
- 4. SoftwareMill
- 5. SoftwareMill
- 7. SoftwareMill ● Hibernate Envers ● Alpakka Kafka ● Sttp ● Scala Clippy ● Quicklens ● MacWire
- 8. What is Event Sourcing? DB Order { items=[itemA, itemB] }
- 9. What is Event Sourcing? DB DB Order { items=[itemA, itemB] } ItemAdded(itemA) ItemAdded(itemC) ItemRemoved(itemC) ItemAdded(itemB)
- 10. History ● 9000 BC, Mesopotamian Clay Tablets, e.g. for market transactions
- 11. History ● 2005, Event Sourcing “Enterprise applications that use Event Sourcing are rarer, but I have seen a few applications (or parts of applications) that use it.”
- 12. Why Event Sourcing? ● complete log of every state change ● debugging ● performance ● scalability ● microservices integration pattern
- 13. ES and CQRS Command Service Domain Events Client Query Service Data access Commands Queries Read modelRead modelRead models
- 14. ES and CQRS level 1 Command Service Domain Events Client Query Service Data access Commands Queries Read modelRead modelRead models Transaction
- 15. ES and CQRS level 1 ● Entry-level, synchronous & transactional event sourcing ● slick-eventsourcing
- 16. ES and CQRS level 1 + easy to implement + easy to reason about + 0 eventual consistency - performance - scalability
- 17. ES and CQRS level 2 Command Service Domain Events Client Query Service Data access Commands Queries Read modelRead modelRead models Projector Transaction
- 18. ES and CQRS level 2 +/- performance +/- scalability - eventual consistency - increased events DB load ? - lags
- 19. ES and CQRS level 3 Command Service Domain Events Client Query Service Data access Commands Queries Read modelRead modelRead models Projector Transaction event bus
- 20. ES and CQRS level 3.1 Command Service Domain Events Client Query Service Data access Commands Queries Read modelRead modelRead models Projector event bus Transaction
- 21. ES and CQRS level 3.1.1 Command Service Domain Events Client Query Service Data access Commands Queries Read modelRead modelRead models Projector event bus Transaction
- 22. ES and CQRS level 3.1.1 Command Service Domain Events Client Query Service Data access Commands Queries Read modelRead modelRead models Projector event bus Transaction At-least-once delivery
- 23. ES and CQRS level 3.1.1 Command Service Domain Events Client Query Service Data access Commands Queries Read modelRead modelRead models Projector event bus Transaction
- 24. ES and CQRS level 3.1.1 Command Service Domain Events Client Query Service Data access Commands Queries Read modelRead modelRead models Projector event bus Transaction
- 25. ES and CQRS level 3.1.1 Command Service Domain Events Client Query Service Data access Commands Queries Read modelRead modelRead models Projector event bus Transaction
- 26. ES and CQRS level 3.1.1 Command Service Domain Events Client Query Service Data access Commands Queries Read modelRead modelRead models Projector event bus Transaction
- 27. ES and CQRS level 3.1.1 Command Service Domain Events Client Query Service Data access Commands Queries Read modelRead modelRead models Projector event bus Transaction
- 28. ES and CQRS level 3.1.1 Command Service Domain Events Client Query Service Data access Commands Queries Read modelRead modelRead models Projector event bus Transaction ?
- 29. ES and CQRS level 3.2 Events Client Query Service Data access Commands Queries Read modelRead modelRead models Projector event bus Command Service Domain Command Service Domain Command Service Domain Transaction ` Sharded Cluster
- 30. ES and CQRS level 3.x + performance + scalability - eventual consistency - complex implementation
- 31. ES and CQRS alternatives ● Change Capture Data (CDC) logging instead of event bus? ● event bus instead of DB?
- 32. Not covered but worth to check ● Command Sourcing ● Event Collaboration
- 33. ES implementation? ● custom ● library ● framework
- 34. ES from domain perspective ● commands, events, state ● 2 methods on state ○ process(command: Command): List[Event] ○ apply(event: Event): State
- 35. ES from application perspective ● snapshotting ● fail-over ● recover ● debugging ● sharding ● serialization & schema evolution ● concurrency access ● etc.
- 36. import javax.persistence.*; import java.util.List; @Entity public class Issue { @EmbeddedId private IssueId id; private String name; private IssueStatus status; @OneToMany(cascade = CascadeType.MERGE) private List<IssueComment> comments; ... public void changeStatusTo(IssueStatus newStatus) { if (this.status == IssueStatus.DONE && newStatus == IssueStatus.NEW || this.status == IssueStatus.NEW && newStatus == IssueStatus.DONE) { throw new RuntimeException(String.format("Cannot change issue status from %s to %s", this.status, newStatus)); } this.status = newStatus; } ... }
- 37. import org.axonframework.commandhandling.* import org.axonframework.eventsourcing.* @Aggregate(repository = "userAggregateRepository") public class User { @AggregateIdentifier private UserId userId; private String passwordHash; @CommandHandler public boolean handle(AuthenticateUserCommand cmd) { boolean success = this.passwordHash.equals(hashOf(cmd.getPassword())); if (success) { apply(new UserAuthenticatedEvent(userId)); } return success; } @EventSourcingHandler public void on(UserCreatedEvent event) { this.userId = event.getUserId(); this.passwordHash = event.getPassword(); } private String hashOf(char[] password) { return DigestUtils.sha1(String.valueOf(password)); } }
- 38. import akka.Done import com.lightbend.lagom.scaladsl.* import play.api.libs.json.{Format, Json} import com.example.auction.utils.JsonFormats._ class UserEntity extends PersistentEntity { override def initialState = None override def behavior: Behavior = { case Some(user) => Actions().onReadOnlyCommand[GetUser.type, Option[User]] { case (GetUser, ctx, state) => ctx.reply(state) }.onReadOnlyCommand[CreateUser, Done] { case (CreateUser(name), ctx, state) => ctx.invalidCommand("User already exists") } case None => Actions().onReadOnlyCommand[GetUser.type, Option[User]] { case (GetUser, ctx, state) => ctx.reply(state) }.onCommand[CreateUser, Done] { case (CreateUser(name), ctx, state) => ctx.thenPersist(UserCreated(name))(_ => ctx.reply(Done)) }.onEvent { case (UserCreated(name), state) => Some(User(name)) } } }
- 40. import java.time.Instant import info.ludwikowski.es.user.domain.UserCommand.* import info.ludwikowski.es.user.domain.UserEvent.* import scala.util.{Failure, Success, Try} final case class User (userId: UserId, name: String, email: Email) { def applyEvent(userEvent: UserEvent): Try[User] = ??? //pattern matching def process(userCommand: UserCommand): Try[List[UserEvent]] = ??? //pattern matching } object User { def from(u: UserCreated): User = User(u.userId, u.name, u.email) }
- 41. ES packaging ● snapshotting ● fail-over ● recover ● debugging ● sharding ● serialization & schema evolution ● concurrency access ● etc.
- 42. ES packaging ● domain logic ● domain validation ● 0 ES framework imports
- 43. library vs. framework ● Akka Persistence vs. Lagom ● Akka Persistence Typed?
- 44. Event storage ● file ● RDBMS ● Event Store ● MongoDB ● Kafka ● Cassandra
- 45. Event storage for Akka Persistence ● file ● RDBMS ● Event Store ● MongoDB ● Kafka ● Cassandra
- 46. akka-persistence-jdbc trap val theTag = s"%$tag%" sql""" SELECT "#$ordering", "#$deleted", "#$persistenceId", "#$sequenceNumber", "#$message", "#$tags" FROM ( SELECT * FROM #$theTableName WHERE "#$tags" LIKE $theTag AND "#$ordering" > $theOffset AND "#$ordering" <= $maxOffset ORDER BY "#$ordering" ) WHERE rownum <= $max"""
- 47. akka-persistence-jdbc trap SELECT * FROM events_journal WHERE tags LIKE ‘%some_tag%’;
- 48. Cassandra perfect for ES? ● partitioning by design ● replication by design ● leaderless (no single point of failure) ● optimised for writes (2 nodes = 100 000 tx/s) ● near-linear horizontal scaling
- 49. ScyllaDB ? ● Cassandra without JVM ○ same protocol, SSTable compatibility ● C++ and Seastar lib ● up to 1,000,000 IOPS/node ● not fully supported by Akka Persistence
- 50. Event serialization ● plain text ○ JSON ○ XML ○ YAML ● binary ○ java serialization ○ Avro ○ Protocol Buffers (Protobuf) ○ Thrift ○ Kryo
- 51. Plain text Binary human-readable deserialization required
- 52. Plain text Binary human-readable deserialization required precision issues (JSON IEEE 754, DoS) -
- 53. Plain text Binary human-readable deserialization required precision issues (JSON IEEE 754, DoS) - storage consumption compress
- 54. Plain text Binary human-readable deserialization required precision issues (JSON IEEE 754, DoS) - storage consumption compress slow fast
- 55. Plain text Binary human-readable deserialization required precision issues (JSON IEEE 754, DoS) - storage consumption compress slow fast poor schema evolution support full schema evolution support
- 56. Binary ● java serialization ● Avro ● Protocol Buffers (Protobuf) ● Thrift ● Kryo
- 57. Binary ● java serialization ● Avro ● Protocol Buffers (Protobuf) ● Thrift ● Kryo
- 58. Binary ● java serialization ● Avro ● Protocol Buffers (Protobuf) ● Thrift ● Kryo
- 59. Binary ● java serialization ● Avro ● Protocol Buffers (Protobuf) ● Thrift ● Kryo
- 60. Multi-language support ● Avro ○ C, C++, C#, Go, Haskell, Java, Perl, PHP, Python, Ruby, Scala ● Protocol Buffers (Protobuf) ○ even more than Avro
- 63. Full compatibility Application Events ● backward - V2 can read V1 V1 V2
- 64. ● forward - V2 can read V3 Full compatibility Application Events V1, V2 V2 Application Application V2 V3
- 65. ● forward - V2 can read V3 Full compatibility Events Read modelRead modelRead models Projector V2V3
- 66. Schema evolution - full compatibility Protocol Buffers Avro Add field + (optional) + (default value) Remove field + + (default value) Rename field + + (aliases) https://martin.kleppmann.com/2012/12/05/schema-evolution-in-avro-protocol-buffers-thrift.html
- 67. Protobuf schema management //user-events.proto message UserCreatedEvent { string user_id = 1; string operation_id = 2; int64 created_at = 3; string name = 4; string email = 5; } package user.application UserCreatedEvent( userId: String, operationId: String, createdAt: Long, name: String, email: String )
- 68. Protobuf schema management package user.domain UserCreated( userId: UserId, operationId: OperationId, createdAt: Instant, name: String, email: Email ) extends UserEvent package user.application UserCreatedEvent( userId: String, operationId: String, createdAt: Long, name: String, email: String )
- 69. Protobuf schema management ● def toDomain(event: UserCreatedEvent): UserEvent.UserCreated ● def toSerializable(event: UserEvent.UserCreated): UserCreatedEvent
- 70. Protobuf schema management + clean domain - a lot of boilerplate code
- 71. Avro schema management package user.domain UserCreated( userId: UserId, operationId: OperationId, createdAt: Instant, name: String, email: Email ) extends UserEvent { "type" : "record", "name" : "UserCreated", "namespace" : "info.ludwikowski.es.user.domain", "fields" : [ { "name" : "userId", "type" : "string" }, { "name" : "operationId", "type" : "string" }, { "name" : "createdAt", "type" : "long" }, { "name" : "name", "type" : "string" }, { "name" : "email", "type" : "string" } ] }
- 72. Avro deserialization Bytes Deserialization Object Reader SchemaWriter Schema
- 73. Avro writer schema source ● add schema to the payload ● custom solution ○ schema in /resources ○ schema in external storage (must be fault-tolerant) ○ Darwin project ● Schema Registry
- 74. Avro schema management package user.domain UserCreated( userId: UserId, operationId: OperationId, createdAt: Instant, name: String, email: Email ) extends UserEvent { "type" : "record", "name" : "UserCreated", "namespace" : "info.ludwikowski.es.user.domain", "fields" : [ { "name" : "userId", "type" : "string" }, { "name" : "operationId", "type" : "string" }, { "name" : "createdAt", "type" : "long" }, { "name" : "name", "type" : "string" }, { "name" : "email", "type" : "string" } ] }
- 75. Protocol Buffers vs. Avro { "type" : "record", "name" : "UserCreated", "namespace" : "info.ludwikowski.es.user.domain", "fields" : [ { "name" : "userId", "type" : "string" }, { "name" : "operationId", "type" : "string" }, { "name" : "createdAt", "type" : "long" }, { "name" : "name", "type" : "string" }, { "name" : "email", "type" : "string" } ] } message UserCreatedEvent { string user_id = 1; string operation_id = 2; int64 created_at = 3; string name = 4; string email = 5; }
- 76. Avro schema management + less boilerplate code +/- clean domain - reader & writer schema distribution
- 77. Avro + less boilerplate code +/- clean domain - reader & writer schema distribution Protobuf + clean domain - a lot of boilerplate code
- 78. Avro vs. Protocol Buffers ● The best serialization strategy for Event Sourcing
- 83. Immutable vs. mutable state? ● add/remove ImmutableList 17.496 ops/s ● add/remove TreeMap 2201.731 ops/s
- 85. Fixing state ● healing command
- 86. Updating all aggregates User(id)Command(user_id) Event(user_id)Event(user_id)Event(user_id)
- 90. Handling duplicates Events Read modelRead modelRead models Projector event bus idempotent updates
- 91. Event + seq_noEvent + seq_no Handling duplicates Events Read modelRead modelRead models Projector event bus Event + seq_no read model update + seq_no
- 92. Broken read model Events ad model ead model Read models Projector event bus
- 93. Broken read model Events ad model ead model Read models Projector event bus read model update + offset (manual offset management)
- 94. Multi aggregate transactional update ● rethink aggregates boundaries ● compensating action ○ optimistic ○ pessimistic
- 95. Pessimistic compensation action User account Cinema hall
- 96. Pessimistic compensation action User account Cinema hall charged
- 97. Pessimistic compensation action User account Cinema hall charged booked
- 98. Pessimistic compensation action User account Cinema hall charged booked sold out
- 99. Pessimistic compensation action User account Cinema hall charged booked booked sold out
- 100. Pessimistic compensation action User account Cinema hall charged booked booked sold out
- 101. Pessimistic compensation action User account Cinema hall charged booked booked sold out refund
- 102. Optimistic compensation action User account Cinema hall charged booked sold out
- 103. Optimistic compensation action User account Cinema hall booked booked sold out overbooked
- 104. Optimistic compensation action User account Cinema hall charged booked booked sold out overbooked ?
- 105. Saga Command Service Domain Events Client Query Service Data access Commands Queries Read modelRead modelRead models Updater event bus Transaction
- 106. Saga Command Service Domain Events Query Service Data access Commands Queries Read modelRead modelRead models Projector event bus Transaction
- 107. Saga Command Service Domain Events Client Query Service Data access Commands Queries Read modelRead modelRead models Projector event bus Transaction
- 108. Saga ● should be persistable ● events order should be irrelevant ● time window limitation ● compensating action must be commutative
- 109. Saga ● Sagas with ES ● DDD, Saga & Event-sourcing ● Applying Saga Pattern ● Microservice Patterns
- 110. ES with RODO/GDPR ● “right to forget” with: ○ data shredding (and/or deleting) ■ events, state, views, read models ○ retention policy ■ message brokers, backups, logs ○ data before RODO migration
- 111. ES and CQRS level 3.2 Events Client Query Service Data access Commands Queries Read modelRead modelRead models Projector event bus Command Service Domain Command Service Domain Command Service Domain Transaction Sharding Clustering
- 112. Cluster = split brain 1 5 4 3 Load balancer 2
- 113. Cluster = split brain 1 5 4 3 Load balancer 2 User(1) Command(1)
- 114. Cluster = split brain 1 5 4 3 Load balancer 2
- 115. Cluster = split brain 1 5 4 3 Load balancer 2 User(1)
- 116. Cluster = split brain 1 5 4 3 Load balancer 2 User(1) Command(1) User(1)
- 117. Cluster = split brain 1 5 4 3 Load balancer 2 User(1) Command(1) User(1) Command(1)
- 118. Cluster best practises ● remember about the split brain ● very good monitoring & alerting ● a lot of failover tests ● cluster also on dev/staging ● keep it as small as possible (code base, number of nodes, etc.)
- 119. Summary ● carefully choose ES lib/framework ● understand event/command/state schema evolution ● eventual consistency is your friend ● scaling is complex ● database inside-out ● log-based processing mindset
- 121. Rate me please :)
- 122. Thank you and Q&A ➔ ➔ ludwikowski.info ➔ github.com/aludwiko ➔ @aludwikowski
- 123. WE WANT YOU
- 124. Want to learn more? ● Akka Typed Fundamentals (Java/Scala) ● Reactive Event Sourcing with Akka (Java/Scala) ● Performance tests with Gatling