Exploring Lightweight Event Sourcing - GOTO Amsterdam 2011

Exploring Lightweight Event SourcingErik Rozendaal <erozendaal@zilverline.com> @erikrozendaal GOTO Amsterdam 2011

Goals• The problem of data persistence• Understand event sourcing• Show why Scala is well-suited as implementation language Exploring lightweight event sourcing 2

ProblemExploring lightweight event sourcing 3

Data must be durable Exploring lightweight event sourcing 4

But currentapplications are lossy Exploring lightweight event sourcing 5

Lossy?Exploring lightweight event sourcing 6

Lossy?• UPDATE invoice WHERE id = 1234 SET total_amount = 230 Exploring lightweight event sourcing 6

Lossy?• UPDATE invoice WHERE id = 1234 SET total_amount = 230 • What happened to the previous order amount? Exploring lightweight event sourcing 6

Lossy?• UPDATE invoice WHERE id = 1234 SET total_amount = 230 • What happened to the previous order amount? • Why was the order amount changed? Exploring lightweight event sourcing 6

Lossy?• UPDATE invoice WHERE id = 1234 SET total_amount = 230 • What happened to the previous order amount? • Why was the order amount changed?• Application behavior is not captured! Exploring lightweight event sourcing 6

Behavior?Exploring lightweight event sourcing 7

Behavior?Status:"Draft" Total:Invoice 0.00 Exploring lightweight event sourcing 7

Behavior?Status: Status: Recipient:"Draft" "Draft" "Erik" Total: Total:Invoice Invoice 0.00 0.00 Exploring lightweight event sourcing 7

Behavior?Status: Status: Recipient: Status: Recipient:"Draft" "Draft" "Erik" "Draft" "Erik" Total: Total: Total:Invoice Invoice Invoice 0.00 0.00 9.95 Item: Invoice Item "Food" Amount: 9.95 Exploring lightweight event sourcing 7

It’s just data mutationStatus: Status: Recipient: Status: Recipient:"Draft" "Draft" "Erik" "Draft" "Erik" Total: Total: Total:Invoice Invoice Invoice 0.00 0.00 9.95 Item: Invoice Item "Food" Amount: 9.95 Exploring lightweight event sourcing 8

It’s just data mutation Status: Recipient: "Draft" "Erik" Total: Invoice 9.95 Item: Invoice Item "Food" Amount: 9.95 Exploring lightweight event sourcing 8

N-TierExploring lightweight event sourcing 9

N-Tier• Presentation, Service, and Data Layers Exploring lightweight event sourcing 9

N-Tier• Presentation, Service, and Data Layers• Shared data model (“domain”) Exploring lightweight event sourcing 9

N-Tier• Presentation, Service, and Data Layers• Shared data model (“domain”)• Heavy use of a single, global, mutable variable (“the database”) Exploring lightweight event sourcing 9

Object-Relational Mapper“Inexperienced programmers love magic becauseit saves their time. Experienced programmers hatemagic because it wastes their time.”– @natpryce Exploring lightweight event sourcing 10

Transactions• Start transaction• SELECT copy of data from database• ORM hydrates objects to give program private copy of data• ORM compares mutated program copy with initial copy to generate UPDATEs• Commit transaction Exploring lightweight event sourcing 11

Performance Tuning Exploring lightweight event sourcing 12

Performance Tuning• Do you know the queries generated by your ORM? Exploring lightweight event sourcing 12

Performance Tuning• Do you know the queries generated by your ORM?• What’s the query execution plan? Exploring lightweight event sourcing 12

Performance Tuning• Do you know the queries generated by your ORM?• What’s the query execution plan?• Optimize reads versus writes Exploring lightweight event sourcing 12

“Domain” Model Exploring lightweight event sourcing 13

“Domain” Model• Presentation layer needs wide access to many parts Exploring lightweight event sourcing 13

“Domain” Model• Presentation layer needs wide access to many parts• Service layer is only interested in subset related to application behavior Exploring lightweight event sourcing 13

“Domain” Model• Presentation layer needs wide access to many parts• Service layer is only interested in subset related to application behavior• ORM tightly couples domain to relational model Exploring lightweight event sourcing 13

“Domain” Model• Presentation layer needs wide access to many parts• Service layer is only interested in subset related to application behavior• ORM tightly couples domain to relational model• High coupling, low cohesion Exploring lightweight event sourcing 13

Domain Model Status: Recipient: "Draft" "Erik" Invoice Total: 9.95 Item: Invoice Item "Food" Amount: 9.95 Model Exploring lightweight event sourcing 14

Domain Model Status: Recipient: "Draft" "Erik" Invoice Total: 9.95 Item: Invoice Item "Food" Amount: 9.95Service Layer Model Exploring lightweight event sourcing 14

Domain Model Status: Recipient: "Draft" "Erik" Invoice Total: 9.95 Item: Invoice Item "Food" Amount: 9.95Service Layer Model Database Exploring lightweight event sourcing 14

Is it any surprise thatwe’re struggling to build modular, maintainable applications? Exploring lightweight event sourcing 15

Domain Driven Design• Domain-Driven Design An approach to software development that suggests that (1) For most software projects, the primary focus should be on the domain and domain logic; and (2) Complex domain designs should be based on a model.• Domain Expert A member of a software project whose ﬁeld is the domain of the application, rather than software development. Not just any user of the software, the domain expert has deep knowledge of the subject.• Ubiquitous Language A language structured around the domain model and used by all team members to connect all the activities of the team with the software. Exploring lightweight event sourcing 16

Event Sourcing• All state changes are explicitly captured using domain events• Capture the intent of the user and the related data• Events represent the outcome of application behavior Exploring lightweight event sourcing 17

Source Control Exploring lightweight event sourcing 18

Source Control Subversion CVS RCSDarcs Git Mercurial BitKeeper SCCS PVCS Exploring lightweight event sourcing 18

Domain Behavior Exploring lightweight event sourcing 19

Domain Behavior generateDraft Invoice Created Exploring lightweight event sourcing 19

Domain Behavior generateDraft Invoice Created apply Status: "Draft" Total: Invoice 0.00 Exploring lightweight event sourcing 19

Domain Behavior generateDraft Invoice Invoice Recipient Created Changed Recipient: "Erik" apply generate Status: "Draft" Total: Invoice 0.00 Exploring lightweight event sourcing 19

Domain Behavior generateDraft Invoice Invoice Recipient Created Changed Recipient: "Erik" apply generate apply Status: Status: Recipient: "Draft" "Draft" "Erik" Total: Total: Invoice Invoice 0.00 0.00 Exploring lightweight event sourcing 19

Domain Behavior generateDraft Invoice Invoice Recipient Invoice Item Created Changed Added Recipient: "Erik" Item: "Food" Item amount: 9.95 Total amount: 9.95 apply generate apply generate Status: Status: Recipient: "Draft" "Draft" "Erik" Total: Total: Invoice Invoice 0.00 0.00 Exploring lightweight event sourcing 19

Domain Behavior generateDraft Invoice Invoice Recipient Invoice Item Created Changed Added Recipient: "Erik" Item: "Food" Item amount: 9.95 Total amount: 9.95 apply generate apply generate apply Status: Status: Recipient: Status: Recipient: "Draft" "Draft" "Erik" "Draft" "Erik" Total: Total: Total: Invoice Invoice Invoice 0.00 0.00 9.95 Item: Invoice Item "Food" Amount: 9.95 Exploring lightweight event sourcing 19

Only the events needto be stored on disk Event Store Draft Invoice Invoice Recipient Invoice Item ....... Created Changed Added Recipient: "Erik" Item: "Food" Item amount: 9.95 Total amount: 9.95 Exploring lightweight event sourcing 20

Reloading from history Exploring lightweight event sourcing 21

Reloading from historyDraft Invoice Created apply Status: "Draft" Total: Invoice 0.00 Exploring lightweight event sourcing 21

Reloading from historyDraft Invoice Invoice Recipient Created Changed Recipient: "Erik" apply apply Status: Status: Recipient: "Draft" "Draft" "Erik" Total: Total: Invoice Invoice 0.00 0.00 Exploring lightweight event sourcing 21

Reloading from historyDraft Invoice Invoice Recipient Invoice Item Created Changed Added Recipient: "Erik" Item: "Food" Item amount: 9.95 Total amount: 9.95 apply apply apply Status: Status: Recipient: Status: Recipient: "Draft" "Draft" "Erik" "Draft" "Erik" Total: Total: Total: Invoice Invoice Invoice 0.00 0.00 9.95 Item: Invoice Item "Food" Amount: 9.95 Exploring lightweight event sourcing 21

YAGNI?Exploring lightweight event sourcing 22

YAGNI?• On the checkout page we’d like to promote products that customers previously removed from their shopping cart. Exploring lightweight event sourcing 22

YAGNI?• On the checkout page we’d like to promote products that customers previously removed from their shopping cart.• Can you tell us how many people removed a product but bought it later anyway? Exploring lightweight event sourcing 22

YAGNI?• On the checkout page we’d like to promote products that customers previously removed from their shopping cart.• Can you tell us how many people removed a product but bought it later anyway?• ... over the past 5 years? ... and how much time passed in-between? Exploring lightweight event sourcing 22

YAGNI?• We have reports of a strange bug, but have not been able to isolate it. Could you look at the production data to ﬁnd out what’s going on? Exploring lightweight event sourcing 23

Is it worth it? Exploring lightweight event sourcing 24

Is it worth it?• Make the event sourcing implementation as simple as possible Exploring lightweight event sourcing 24

Is it worth it?• Make the event sourcing implementation as simple as possible• ... while avoiding the complexities of databases, ORMs, etc. Exploring lightweight event sourcing 24

Is it worth it?• Make the event sourcing implementation as simple as possible• ... while avoiding the complexities of databases, ORMs, etc. • ... unless you want or need them :) Exploring lightweight event sourcing 24

Implementation Exploring lightweight event sourcing 25

Implementation• Events for durability Exploring lightweight event sourcing 25

Implementation• Events for durability• Keep current state in RAM (Memory Image) Exploring lightweight event sourcing 25

Implementation• Events for durability• Keep current state in RAM (Memory Image)• Scala case classes to deﬁne events and immutable data structures Exploring lightweight event sourcing 25

Implementation• Events for durability• Keep current state in RAM (Memory Image)• Scala case classes to deﬁne events and immutable data structures• Independent components composed into a single application Exploring lightweight event sourcing 25

Simple functionality example• “CRUD”: no domain logic, so no aggregate• So we’ll persist events directly from the UI• Useful to get started• ... and even complex applications still contain simple functionality Exploring lightweight event sourcing 26

“CRUD”Exploring lightweight event sourcing 27

“CRUD”Create/Update/Delete Exploring lightweight event sourcing 27

“CRUD”Create/Update/Delete HTTP POST Validate input and generate event Exploring lightweight event sourcing 27

“CRUD”Create/Update/Delete HTTP POST Save Event Validate input and Event generate event Store Exploring lightweight event sourcing 27

“CRUD”Create/Update/Delete HTTP POST Save Event Validate input and Event generate event Store Apply Event Status: Recipient: "Draft" "Erik" Invoice Total: 9.95 Item: Invoice Item "Food" Amount: 9.95 Presentation Model Exploring lightweight event sourcing 27

“CRUD”Create/Update/Delete HTTP POST Save Event Validate input and Event generate event Store Apply Event Status: Recipient: "Draft" "Erik" Invoice Total: 9.95 Query Render View Item: Invoice Item "Food" Amount: 9.95 Presentation Model Exploring lightweight event sourcing 27

“CRUD”Create/Update/Delete HTTP POST Save Event Validate input and Event generate event Store Apply Event Read Status: "Draft" Recipient: "Erik" Invoice Total: 9.95 HTTP GET Query Render View Item: Invoice Item "Food" Amount: 9.95 Presentation Model Exploring lightweight event sourcing 27

HTTP POST Save Event Validate input and EventController generate event Store Apply Event Status: Recipient: "Draft" "Erik" Invoice Total: 9.95 HTTP GET Query Render View Item: Invoice Item "Food" Amount: 9.95post("/todo") { field("text", required) match { case Success(text) => commit(ToDoItemAdded(ToDoItem(UUID.randomUUID(), text))) redirect(url("/todo")) case Failure(error) => new ToDoView(toDoItems, Some(error)), }} Exploring lightweight event sourcing 28

HTTP POST Save Event Validate input and EventMemory Image generate event Store Apply Event Status: Recipient: "Draft" "Erik" Invoice Total: 9.95 HTTP GET Query Render View Item: Invoice Item "Food" Amount: 9.95case class ToDoItems ( all : Map[UUID, ToDoItem] = Map.empty, recentlyAdded: Vector[UUID] = Vector.empty) { def apply(event: ToDoItemEvent) = event match { case ToDoItemAdded(item) => copy(all + (item.id -> item), recentlyAdded :+ item.id) // [... handle other event types ...] } def mostRecent(count: Int) = recentlyAdded.takeRight(count).map(all).reverse} Exploring lightweight event sourcing 29

HTTP POST Save Event Validate input and EventView generate event Store Apply Event Status: Recipient: "Draft" "Erik" Invoice Total: 9.95 HTTP GET Query Render View Item: Invoice Item "Food" Amount: 9.95<table class="zebra-striped"> <thead><tr><th>To-Do</th></tr></thead> <tbody>{ for (item <- toDoItems.mostRecent(20)) yield { <tr><td>{ item.text }</td></tr> } }</tbody></table> Exploring lightweight event sourcing 30

Domain Driven Design HTTP POST Save Event Validate input and Event generate event Store Apply Event Read Status: "Draft" Recipient: "Erik" Invoice Total: 9.95 HTTP GET Query Render View Item: Invoice Item "Food" Amount: 9.95 Exploring lightweight event sourcing 31

Domain Driven Design Domain ModelHTTP POST Command Save Event Validate input and Event generate event Store Apply Event Read Status: "Draft" Recipient: "Erik" Invoice Total: 9.95 HTTP GET Query Render View Item: Invoice Item "Food" Amount: 9.95 Exploring lightweight event sourcing 31

Domain Codesealed trait InvoiceEventcase class InvoiceCreated() extends InvoiceEventcase class InvoiceRecipientChanged(recipient: String) extends InvoiceEventcase class InvoiceItemAdded(item: InvoiceItem, totalAmount: BigDecimal) extends InvoiceEventcase class InvoiceSent(sentOn: LocalDate, paymentDueOn: LocalDate) extends InvoiceEvent Exploring lightweight event sourcing 32

Domain Codecase class DraftInvoice( recipient: Option[String] = None, nextItemId: Int = 1, items: Map[Int, InvoiceItem] = Map.empty) extends Invoice { def addItem(description: String, amount: BigDecimal): Behavior[DraftInvoice] = ... // [... other domain logic ...] private def itemAdded = when[InvoiceItemAdded] { event => // ... }} Exploring lightweight event sourcing 33

Domain Codecase class DraftInvoice( recipient: Option[String] = None, nextItemId: Int = 1, The “Brains” items: Map[Int, InvoiceItem] = Map.empty) extends Invoice { def addItem(description: String, amount: BigDecimal): Behavior[DraftInvoice] = ... // [... other domain logic ...] private def itemAdded = when[InvoiceItemAdded] { event => // ... }} Exploring lightweight event sourcing 33

Domain Codecase class DraftInvoice( recipient: Option[String] = None, nextItemId: Int = 1, The “Brains” items: Map[Int, InvoiceItem] = Map.empty) extends Invoice { def addItem(description: String, amount: BigDecimal): Behavior[DraftInvoice] = ... // [... other domain logic ...] The “Muscle” private def itemAdded = when[InvoiceItemAdded] { event => // ... }} Exploring lightweight event sourcing 33

Domain Codecase class DraftInvoice( recipient: Option[String] = None, nextItemId: Int = 1, items: Map[Int, InvoiceItem] = Map.empty) extends Invoice { // ... def addItem(description: String, amount: BigDecimal): Behavior[DraftInvoice] = itemAdded(InvoiceItemAdded(InvoiceItem(nextItemId, description, amount), totalAmount + amount)) private def totalAmount = items.values.map(_.amount).sum private def readyToSend_? = recipient.isDefined && items.nonEmpty // ... private def itemAdded = when[InvoiceItemAdded] { event => copy(nextItemId = nextItemId + 1, items = items + (event.item.id -> event.item)) }} Exploring lightweight event sourcing 34

Domain Codecase class DraftInvoice( recipient: Option[String] = None, nextItemId: Int = 1, items: Map[Int, InvoiceItem] = Map.empty) extends Invoice { ... /** Reload from history. */ protected[this] def applyEvent = recipientChanged orElse itemAdded orElse sent ...} Exploring lightweight event sourcing 35

Cooperating UsersCreate/Update/Delete HTTP POST Save Event Validate input and Event generate event Store Apply Event Read Status: "Draft" Recipient: "Erik" Invoice Total: 9.95 HTTP GET Query Render View Item: Invoice Item "Food" Amount: 9.95 Exploring lightweight event sourcing 36

Cooperating Users Submit EventHTTP POST Save Event Validate input and Event Conﬂict Resolution generate event Store Conﬂicting Events Apply Event Read Status: "Draft" Recipient: "Erik" Invoice Total: 9.95 HTTP GET Query Render View Item: Invoice Item "Food" Amount: 9.95 Exploring lightweight event sourcing 36

Cooperating Users Submit EventHTTP POST Save Event Validate input and Event Conﬂict Resolution generate event Store Conﬂicting Events Apply Event Read Checkout revision 23 Status: "Draft" Recipient: "Erik" Invoice Total: 9.95 HTTP GET Query Render View Item: Invoice Item "Food" Amount: 9.95 Exploring lightweight event sourcing 36

Cooperating Users Submit events based on revision 23 Submit EventHTTP POST Save Event Validate input and Event Conﬂict Resolution generate event Store Conﬂicting Events Apply Event Read Checkout revision 23 Status: "Draft" Recipient: "Erik" Invoice Total: 9.95 HTTP GET Query Render View Item: Invoice Item "Food" Amount: 9.95 Exploring lightweight event sourcing 36

Cooperating Users Compare submitted events Submit events with intermediate based on revision 23 events baed on current revision Submit EventHTTP POST Save Event Validate input and Event Conﬂict Resolution generate event Store Conﬂicting Events Apply Event Read Checkout revision 23 Status: "Draft" Recipient: "Erik" Invoice Total: 9.95 HTTP GET Query Render View Item: Invoice Item "Food" Amount: 9.95 Exploring lightweight event sourcing 36

Events Event Store Status: Recipient: "Draft" "Erik" Invoice Total: 9.95 Item: Invoice Item "Food" Amount: 9.95 Presentation ModelExploring lightweight event sourcing 37

Events Event StorePush Status: Recipient: "Draft" "Erik" Invoice Total: 9.95 Item: Invoice Item "Food" Amount: 9.95 Presentation Model Exploring lightweight event sourcing 37

Events Event StorePush Status: Recipient: "Draft" "Erik" Invoice Total: 9.95 Item: Invoice Item "Food" RDBMS Amount: 9.95 Presentation Model Exploring lightweight event sourcing 37

Events Event StorePush Status: Recipient: "Draft" "Erik" Invoice Total: 9.95 Item: Invoice Item "Food" RDBMS Solr / Neo4J / HBase / ... Amount: 9.95 Presentation Model Exploring lightweight event sourcing 37

Events Event StorePush System Integration Order Fullﬁllment Payment Provider Status: Recipient: "Draft" "Erik" Legacy System Invoice Total: 9.95 Item: Invoice Item "Food" RDBMS Solr / Neo4J / HBase / ... Amount: 9.95 Presentation Model Exploring lightweight event sourcing 37

Events Event Store ReplicaPush System Integration Order Fullﬁllment Payment Provider Status: Recipient: "Draft" "Erik" Legacy System Invoice Total: 9.95 Item: Invoice Item "Food" RDBMS Solr / Neo4J / HBase / ... Amount: 9.95 Presentation Model Exploring lightweight event sourcing 37

Events Auditors Event Store ReplicaPush System Integration Order Fullﬁllment Payment Provider Status: Recipient: "Draft" "Erik" Legacy System Invoice Total: 9.95 Item: Invoice Item "Food" RDBMS Solr / Neo4J / HBase / ... Amount: 9.95 Presentation Model Exploring lightweight event sourcing 37

Event Store• Stores sequence of events• Dispatches events when successfully committed• Replays events on startup• It’s your application’s transaction log Exploring lightweight event sourcing 38

Writing Events to Diskprivate class JournalFileWriter(file: File, var sequence: Long) { private val checksum = new CRC32() private val fileOutputStream = new FileOutputStream(file) private val dataOutputStream = new DataOutputStream( new CheckedOutputStream( new BufferedOutputStream(fileOutputStream), checksum)) def write(commit: Array[Byte]) { sequence += 1 checksum.reset() dataOutputStream.writeLong(sequence) dataOutputStream.writeInt(commit.size) dataOutputStream.write(commit) dataOutputStream.writeInt(checksum.getValue().toInt) } def sync(metadata: Boolean = false) { dataOutputStream.flush() fileOutputStream.getChannel().force(metadata) }} Exploring lightweight event sourcing 39

Example Application• https://github.com/erikrozendaal/scala- event-sourcing-example• (soon) https://github.com/zilverline/lessdb- example • Blog series coming soon at http:// blog.zilverline.com Exploring lightweight event sourcing 40

Implementation Limitations• Single-threaded event store using disruptor pattern • ~ 5.000 commits per second• Number of events to replay on startup • ~ 70.000 JSON serialized events/second • ~ 200.000 protobuf serialized events/second• Total number of objects to store in main memory • JVM can run with large heaps (tens of gigabytes) Exploring lightweight event sourcing 41

But ready to scale• Advanced event store implementations support SQL, MongoDB, Amazon SimpleDB, etc.• Use persisted view model for high volume objects (ﬁxes startup time and memory usage)• Easy partitioning of aggregates (consistency boundary with unique identiﬁer)• Load aggregates on-demand, use snapshotting Exploring lightweight event sourcing 42

Conclusion• Fully capture historical information• Independent components where each part of the application can use its own data model• Easier to fully understand compared to traditional ORM based approach• Need to learn “Event-Driven” thinking Exploring lightweight event sourcing 43

Thanks!

References• Example code https://github.com/erikrozendaal/scala-event-sourcing-example• CQRS http://cqrsinfo.com/• Greg Young, “Unshackle Your Domain” http://www.infoq.com/presentations/greg-young-unshackle-qcon08• Pat Helland, “Life Beyond Distributed Transactions: An Apostates Opinion” http://www.ics.uci.edu/~cs223/papers/cidr07p15.pdf• Erik Rozendaal, “Towards an immutable domain model” http:// blog.zilverline.com/2011/02/10/towards-an-immutable-domain-model- monads-part-5/• Martin Fowler, “Memory Image”, http://martinfowler.com/bliki/ MemoryImage.html• Martin Fowler, “The LMAX Architecture”, http://martinfowler.com/articles/ lmax.html Exploring lightweight event sourcing 45

Exploring Lightweight Event Sourcing - GOTO Amsterdam 2011

by Erik Rozendaal

Accessibility

Categories

Upload Details

Usage Rights

2 Embeds 7

Statistics