Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Event sourcing - what could go wrong?

869 views

Published on

Yet another presentation about Event Sourcing? Yes and no. Event Sourcing is a really great concept. Some could say it’s a Holy Grail of the software architecture. I might agree with that, while remembering that everything comes with a price. This session is a summary of my experience with ES gathered while working on 3 different commercial products. Instead of theoretical aspects, I will focus on possible challenges with ES implementation. What could explode (very often with delayed ignition)? How and where to store events effectively? What are possible schema evolution solutions? How to achieve the highest level of scalability and live with eventual consistency? And many other interesting topics that you might face when experimenting with ES.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Event sourcing - what could go wrong?

  1. 1. Event Sourcing - what could go wrong? Andrzej Ludwikowski
  2. 2. About me ➔ ➔ aludwikowski.blogspot.com ➔ github.com/aludwiko ➔ @aludwikowski
  3. 3. What is Event Sourcing? DB Order { items=[itemA, itemB] }
  4. 4. What is Event Sourcing? DB DB Order { items=[itemA, itemB] } ItemAdded(itemA) ItemAdded(itemC) ItemRemoved(itemC) ItemAdded(itemB)
  5. 5. History
  6. 6. History ● 9000 BC, Mesopotamian Clay Tablets, e.g. for market transactions
  7. 7. History ● 2005, Event Sourcing “Enterprise applications that use Event Sourcing are rarer, but I have seen a few applications (or parts of applications) that use it.”
  8. 8. Why Event Sourcing? ● complete log of every state change ● debugging ● performance ● scalability
  9. 9. ES and CQRS Command Service Domain Events Client Query Service Data access Commands Queries Read modelRead modelRead models
  10. 10. ES and CQRS level 1 Command Service Domain Events Client Query Service Data access Commands Queries Read modelRead modelRead models Transaction
  11. 11. ES and CQRS level 1 ● Entry-level, synchronous & transactional event sourcing ● slick-eventsourcing
  12. 12. ES and CQRS level 1 + easy to implement + easy to reason about + 0 eventual consistency - performance - scalability
  13. 13. ES and CQRS level 2 Command Service Domain Events Client Query Service Data access Commands Queries Read modelRead modelRead models Updater Transaction
  14. 14. ES and CQRS level 2 +/- performance +/- scalability - eventual consistency - increased events DB load - lags
  15. 15. ES and CQRS level 3 Command Service Domain Events Client Query Service Data access Commands Queries Read modelRead modelRead models Updater Transaction event bus
  16. 16. ES and CQRS level 3.1 Command Service Domain Events Client Query Service Data access Commands Queries Read modelRead modelRead models Updater event bus Transaction
  17. 17. ES and CQRS level 3.1.1 Command Service Domain Events Client Query Service Data access Commands Queries Read modelRead modelRead models Updater event bus Transaction
  18. 18. ES and CQRS level 3.1.1 Command Service Domain Events Client Query Service Data access Commands Queries Read modelRead modelRead models Updater event bus Transaction At-least-once delivery
  19. 19. ES and CQRS level 3.1.1 Command Service Domain Events Client Query Service Data access Commands Queries Read modelRead modelRead models Updater event bus Transaction
  20. 20. ES and CQRS level 3.1.1 Command Service Domain Events Client Query Service Data access Commands Queries Read modelRead modelRead models Updater event bus Transaction
  21. 21. ES and CQRS level 3.1.1 Command Service Domain Events Client Query Service Data access Commands Queries Read modelRead modelRead models Updater event bus Transaction
  22. 22. ES and CQRS level 3.1.1 Command Service Domain Events Client Query Service Data access Commands Queries Read modelRead modelRead models Updater event bus Transaction
  23. 23. ES and CQRS level 3.1.1 Command Service Domain Events Client Query Service Data access Commands Queries Read modelRead modelRead models Updater event bus Transaction
  24. 24. ES and CQRS level 3.1.1 Command Service Domain Events Client Query Service Data access Commands Queries Read modelRead modelRead models Updater event bus Transaction ?
  25. 25. ES and CQRS level 3.2 Events Client Query Service Data access Commands Queries Read modelRead modelRead models Updater event bus Command Service Domain Command Service Domain Command Service Domain Transaction Sharded Cluster
  26. 26. ES and CQRS level 3.x + performance + scalability - eventual consistency - complex implementation
  27. 27. ES and CQRS alternatives ● Change Capture Data (CDC) logging instead of message queue? ● message queue instead of DB?
  28. 28. ES implementation? ● custom ● library ● framework
  29. 29. ES from domain perspective ● commands, events, state ● 2 main methods on state ○ process(command: Command): List[Event] ○ apply(event: Event): State
  30. 30. ES from application perspective ● snapshotting ● fail-over ● recover ● debugging ● sharding ● serialization & schema evolution ● concurrency access ● etc.
  31. 31. import javax.persistence.*; import java.util.List; @Entity public class Issue { @EmbeddedId private IssueId id; private String name; private IssueStatus status; @OneToMany(cascade = CascadeType.MERGE) private List<IssueComment> comments; ... public void changeStatusTo(IssueStatus newStatus) { if (this.status == IssueStatus.DONE && newStatus == IssueStatus.NEW || this.status == IssueStatus.NEW && newStatus == IssueStatus.DONE) { throw new RuntimeException(String.format("Cannot change issue status from %s to %s", this.status, newStatus)); } this.status = newStatus; } ... }
  32. 32. import org.axonframework.commandhandling.* import org.axonframework.eventsourcing.* @Aggregate(repository = "userAggregateRepository") public class User { @AggregateIdentifier private UserId userId; private String passwordHash; @CommandHandler public boolean handle(AuthenticateUserCommand cmd) { boolean success = this.passwordHash.equals(hashOf(cmd.getPassword())); if (success) { apply(new UserAuthenticatedEvent(userId)); } return success; } @EventSourcingHandler public void on(UserCreatedEvent event) { this.userId = event.getUserId(); this.passwordHash = event.getPassword(); } private String hashOf(char[] password) { return DigestUtils.sha1(String.valueOf(password)); } }
  33. 33. import akka.Done import com.lightbend.lagom.scaladsl.* import play.api.libs.json.{Format, Json} import com.example.auction.utils.JsonFormats._ class UserEntity extends PersistentEntity { override def initialState = None override def behavior: Behavior = { case Some(user) => Actions().onReadOnlyCommand[GetUser.type, Option[User]] { case (GetUser, ctx, state) => ctx.reply(state) }.onReadOnlyCommand[CreateUser, Done] { case (CreateUser(name), ctx, state) => ctx.invalidCommand("User already exists") } case None => Actions().onReadOnlyCommand[GetUser.type, Option[User]] { case (GetUser, ctx, state) => ctx.reply(state) }.onCommand[CreateUser, Done] { case (CreateUser(name), ctx, state) => ctx.thenPersist(UserCreated(name))(_ => ctx.reply(Done)) }.onEvent { case (UserCreated(name), state) => Some(User(name)) } } }
  34. 34. ES packaging
  35. 35. import java.time.Instant import info.ludwikowski.es.user.domain.UserCommand.* import info.ludwikowski.es.user.domain.UserEvent.* import scala.util.{Failure, Success, Try} final case class User private[domain] (userId: UserId, createdAt: Instant, name: String, email: Email) { def applyEvent(userEvent: UserEvent): Try[User] = ??? //pattern matching def process(userCommand: UserCommand): Try[List[UserEvent]] = ??? //pattern matching } object User { def from(u: UserCreated): User = User(u.userId, u.createdAt, u.name, u.email) }
  36. 36. ES packaging ● snapshotting ● fail-over ● recover ● debugging ● sharding ● serialization & schema evolution ● concurrency access ● etc.
  37. 37. ES packaging ● domain logic ● domain validation ● 0 framework/library imports
  38. 38. library vs. framework ● Akka Persistence vs. Lagom
  39. 39. Event storage ● file ● RDBMS ● Event Store ● MongoDB ● Kafka ● Cassandra
  40. 40. Event storage for Akka Persistence ● file ● RDBMS ● Event Store ● MongoDB ● Kafka ● Cassandra
  41. 41. akka-persistence-jdbc trap val theTag = s"%$tag%" sql""" SELECT "#$ordering", "#$deleted", "#$persistenceId", "#$sequenceNumber", "#$message", "#$tags" FROM ( SELECT * FROM #$theTableName WHERE "#$tags" LIKE $theTag AND "#$ordering" > $theOffset AND "#$ordering" <= $maxOffset ORDER BY "#$ordering" ) WHERE rownum <= $max"""
  42. 42. akka-persistence-jdbc trap SELECT * FROM events_journal WHERE tags LIKE ‘%some_tag%’;
  43. 43. Cassandra perfect for ES? ● partitioning by design ● replication by design ● leaderless (no single point of failure) ● optimised for writes (2 nodes = 100 000 tx/s) ● near-linear horizontal scaling
  44. 44. ScyllaDB ? ● Cassandra without JVM ○ same protocol, SSTable compatibility ● C++ and Seastar lib ● 1,000,000 IOPS ● not fully supported by Akka Persistence
  45. 45. Event serialization ● plain text ○ JSON ○ XML ○ YAML ● binary ○ java serialization ○ Avro ○ Protocol Buffers (Protobuf) ○ Thrift ○ Kryo
  46. 46. Plain text Binary human-readable deserialization required
  47. 47. Plain text Binary human-readable deserialization required problems with precision (JSON IEEE 754) -
  48. 48. Plain text Binary human-readable deserialization required precision issues (JSON IEEE 754, DoS) - storage consumption compress
  49. 49. Plain text Binary human-readable deserialization required problems with precision (JSON IEEE 754) - storage consumption compress slow fast
  50. 50. Plain text Binary human-readable deserialization required problems with precision (JSON IEEE 754) - storage consumption compress slow fast poor schema evolution support full schema evolution support
  51. 51. Binary ● java serialization ● Avro ● Protocol Buffers (Protobuf) ● Thrift ● Kryo
  52. 52. Binary ● java serialization ● Avro ● Protocol Buffers (Protobuf) ● Thrift ● Kryo
  53. 53. Binary ● java serialization ● Avro ● Protocol Buffers (Protobuf) ● Thrift ● Kryo
  54. 54. Binary ● java serialization ● Avro ● Protocol Buffers (Protobuf) ● Thrift ● Kryo
  55. 55. Multi-language support ● Avro ○ C, C++, C#, Go, Haskell, Java, Perl, PHP, Python, Ruby, Scala ● Protocol Buffers (Protobuf) ○ even more than Avro
  56. 56. Speed https://code.google.com/archive/p/thrift-protobuf-compare/wikis/Benchmarking.wiki
  57. 57. Size https://code.google.com/archive/p/thrift-protobuf-compare/wikis/Benchmarking.wiki
  58. 58. Full compatibility Application Events ● backward - V2 can read V1 V1 V2
  59. 59. ● forward - V2 can read V3 Full compatibility Application Events V1, V2 V2 Application Application V2 V3
  60. 60. Schema evolution - full compatibility Protocol Buffers Avro Add field + (optional) + (default value) Remove field + + (default value) Rename field + + (aliases) https://martin.kleppmann.com/2012/12/05/schema-evolution-in-avro-protocol-buffers-thrift.html
  61. 61. Protobuf schema management //user-events.proto message UserCreatedEvent { string user_id = 1; string operation_id = 2; int64 created_at = 3; string name = 4; string email = 5; } package user.application UserCreatedEvent( userId: String, operationId: String, createdAt: Long, name: String, email: String )
  62. 62. Protobuf schema management package user.domain UserCreated( userId: UserId, operationId: OperationId, createdAt: Instant, name: String, email: Email ) extends UserEvent package user.application UserCreatedEvent( userId: String, operationId: String, createdAt: Long, name: String, email: String )
  63. 63. Protobuf schema management ● def toDomain(event: UserCreatedEvent): UserEvent.UserCreated ● def toSerializable(event: UserEvent.UserCreated): UserCreatedEvent
  64. 64. Protobuf schema management + clean domain - a lot of boilerplate code
  65. 65. Avro schema management package user.domain UserCreated( userId: UserId, operationId: OperationId, createdAt: Instant, name: String, email: Email ) extends UserEvent { "type" : "record", "name" : "UserCreated", "namespace" : "info.ludwikowski.es.user.domain", "fields" : [ { "name" : "userId", "type" : "string" }, { "name" : "operationId", "type" : "string" }, { "name" : "createdAt", "type" : "long" }, { "name" : "name", "type" : "string" }, { "name" : "email", "type" : "string" } ] }
  66. 66. Avro deserialization Bytes Deserialization Object Reader SchemaWriter Schema
  67. 67. Avro writer schema source ● add schema to the payload ● custom solution ○ schema in /resources ○ schema in external storage (must be fault-tolerant) ● Schema Registry
  68. 68. Avro schema management package user.domain UserCreated( userId: UserId, operationId: OperationId, createdAt: Instant, name: String, email: Email ) extends UserEvent { "type" : "record", "name" : "UserCreated", "namespace" : "info.ludwikowski.es.user.domain", "fields" : [ { "name" : "userId", "type" : "string" }, { "name" : "operationId", "type" : "string" }, { "name" : "createdAt", "type" : "long" }, { "name" : "name", "type" : "string" }, { "name" : "email", "type" : "string" } ] }
  69. 69. Protocol Buffers vs. Avro { "type" : "record", "name" : "UserCreated", "namespace" : "info.ludwikowski.es.user.domain", "fields" : [ { "name" : "userId", "type" : "string" }, { "name" : "operationId", "type" : "string" }, { "name" : "createdAt", "type" : "long" }, { "name" : "name", "type" : "string" }, { "name" : "email", "type" : "string" } ] } message UserCreatedEvent { string user_id = 1; string operation_id = 2; int64 created_at = 3; string name = 4; string email = 5; }
  70. 70. Avro schema management + less boilerplate code +/- clean domain - reader & writer schema distribution
  71. 71. Memory consumption
  72. 72. Immutable vs. mutable state? ● add/remove ImmutableList 17.496 ops/s ● add/remove TreeMap 2201.731 ops/s
  73. 73. Fixing state ● healing command
  74. 74. Updating all aggregates User(id)Command(user_id) Event(user_id)Event(user_id)Event(user_id)
  75. 75. Event + seq_noEvent + seq_no Handling duplicates Events Read modelRead modelRead models Updater event bus Event + seq_no At-least-once delivery
  76. 76. Broken read model Events ad model ead model Read models Updater event bus
  77. 77. Broken read model Events ad model ead model Read models Updater event bus read model update + offset
  78. 78. ES and CQRS level 3.2 Events Client Query Service Data access Commands Queries Read modelRead modelRead models Updater event bus Command Service Domain Command Service Domain Command Service Domain Transaction Sharding Clustering
  79. 79. Cluster = split brain 1 5 4 3 Load balancer 2
  80. 80. Cluster = split brain 1 5 4 3 Load balancer 2 User(1) Command(1)
  81. 81. Cluster = split brain 1 5 4 3 Load balancer 2 User(1)
  82. 82. Cluster = split brain 1 5 4 3 Load balancer 2 User(1) Command(1) User(1)
  83. 83. Cluster = split brain 1 5 4 3 Load balancer 2 User(1) Command(1) User(1) Command(1)
  84. 84. Cluster best practises ● very good monitoring & alerting ● a lot of failover tests ● cluster also on dev/staging ● keep it as small as possible (code base, number of nodes, etc.)
  85. 85. Summary ● understand event/state schema evolution ● eventual consistency is your friend ● scaling is complex ● log-based processing mindset
  86. 86. WE WANT YOU
  87. 87. About me ➔ ➔ aludwikowski.blogspot.com ➔ github.com/aludwiko ➔ @aludwikowski

×