Downloaded 40 times
Uploaded bySteve Pember
Richer data-history-event-sourcing
The document discusses event sourcing as a data storage pattern that captures facts and maintains a complete historical record of events related to aggregates, promoting additivity and losslessness. It addresses the challenges of implementing event sourcing, including increased CPU overhead and storage requirements, while emphasizing its benefits like simplified debugging and business analysis. Additionally, it covers implementation theories, persistence options, querying events, and suggests libraries for event storage.
More Related Content
Similar to Richer data-history-event-sourcing
![Coded Agents – with UiPath SDK + LangGraph [Virtual Hands-on Workshop]](https://cdn.slidesharecdn.com/ss_thumbnails/codedagentsdeck-251215155422-5497c599-thumbnail.jpg?width=640&height=640&fit=bounds)
Richer data-history-event-sourcing
- 1.
- 2. First, A QuickScenario
- 5.
- 6.
- 7.
- 8.
- 10.
- 11.
- 12. THIRDCHANNEL Richer Data HistoryWith Event Sourcing Steve Pember CTO, ThirdChannel
- 13. THIRDCHANNEL Agenda • Event Sourcing •Event Sourcing Challenges • Don’t Worry, ES Is Worth It • Implementation (Theory) • Implementation (Reality) • Event Storage • Querying • Demo
- 14. What Is EventSourcing?
- 15. THIRDCHANNEL Event Sourcing • AlternativeStorage Pattern
- 16.
- 17.
- 19.
- 20.
- 21. THIRDCHANNEL Event Sourcing • AlternativeStorage Pattern • Series of Facts • Aggregates << Events
- 22. Event: Something WhichHas Occurred In The System
- 23. Aggregate: The Objectof Events’ Affections
- 24. THIRDCHANNEL Event Sourcing • AlternativeStorage Pattern • Series of Facts • Aggregates << Events • Purely Additive
- 25. There is NoDelete
- 26.
- 27. Full History ofthe Aggregates
- 28.
- 29. THIRDCHANNEL Event Sourcing • AlternativeStorage Pattern • Series of Facts • Aggregates << Events • Purely Additive • Optimization: Snapshots
- 30.
- 31.
- 32. THIRDCHANNEL Event Sourcing • AlternativeStorage Pattern • Series of Facts • Aggregates << Events • Purely Additive • Optimization: Snapshots • Example: Shopping Cart
- 33.
- 34.
- 35.
- 36. THIRDCHANNEL Event Sourcing • AlternativeStorage Pattern • Series of Facts • Aggregates << Events • Purely Additive • Optimization: Snapshots • Example: Shopping Cart • Transient Objects
- 38. Objects are TransientDerivatives of Your Events
- 39. THIRDCHANNEL Event Sourcing • AlternativeStorage Pattern • Series of Facts • Aggregates << Events • Purely Additive • Optimization: Snapshots • Example: Shopping Cart • Transient Objects • Natural
- 41.
- 42.
- 43. THIRDCHANNEL Questions So Far? •Events & Aggregates • Snapshots • Additive, loss-less model • Transient Derivatives
- 44. THIRDCHANNEL Agenda • Event Sourcing •Event Sourcing Challenges
- 45. “I Think IGet What You’re Laying Down, But It Still Seems Bananas”
- 47.
- 48. “That’s gotta bea ton of CPU overhead”
- 49. “Where’d my Modelsgo? I can’t live without my MVC!”
- 50. THIRDCHANNEL Event Sourcing Challenges •Additional Work To Apply • Additional CPU Processing • Non-Traditional Concept of Models
- 51. THIRDCHANNEL Event Sourcing Challenges •Additional Work To Apply • Additional CPU Processing • Non-Traditional Concept of Models • More Storage Required VS non-ES
- 52.
- 53. THIRDCHANNEL Event Sourcing Challenges •Additional Work To Apply • Additional CPU Processing • Non-Traditional Concept of Models • More Storage Required VS non-ES • Reduced Database Level Constraints
- 54. Make Use ofTransactional Code
- 55. THIRDCHANNEL Event Sourcing Challenges •Additional Work To Apply • Additional CPU Processing • Non-Traditional Concept of Models • More Storage Required VS non-ES • Reduced Transactions / Database Level Constraints • May Be Confusing For Junior Engineers
- 56.
- 58.
- 59. So Why UseES?
- 60. THIRDCHANNEL Agenda • Event Sourcing •Event Sourcing Challenges • Don’t Worry, ES Is Worth It
- 61. THIRDCHANNEL Why Event Sourcing •More Than an Audit Log
- 62. Audit Logs Tellthe History Events Tell the Intent of History
- 64. ES is alsoa perfect Audit Log
- 65. THIRDCHANNEL Why Event Sourcing •More Than an Audit Log • Data Storage is Inexpensive
- 66. THIRDCHANNEL Why Event Sourcing •More Than an Audit Log • Data Storage is Inexpensive • Used By All Long-Running Businesses
- 67.
- 69. All Business ProblemsCan Be Modeled With Event Sourcing
- 70. THIRDCHANNEL Why Event Sourcing •More Than an Audit Log • Data Storage is Inexpensive • Used By All Long-Running Businesses • Only Structural Model That Does Not Lose Information
- 71. Remember: There IsNo Delete
- 72. THIRDCHANNEL Why Event Sourcing •More Than an Audit Log • Data Storage is Inexpensive • Used By All Long-Running Businesses • Only Structural Model That Does Not Lose Information • Simplified Testing and Debugging
- 73. Unit Test Events,Assert In Integration Tests
- 74. Time Travel EasesDebugging
- 75.
- 76.
- 77.
- 78.
- 79.
- 80.
- 81. THIRDCHANNEL Why Event Sourcing •More Than an Audit Log • Data Storage is Inexpensive • Used By All Long-Running Businesses • Only Structural Model That Does Not Lose Information • Simplified Testing and Debugging • Ideal for Business Analysis
- 82.
- 83.
- 85. But With EventSourcing…
- 86. –Johnny Appleseed “Type aquote here.”
- 87.
- 88. Current State isa Projection
- 89.
- 90.
- 91.
- 92.
- 93. THIRDCHANNEL Have you everbuilt? • Versioning of objects • An ‘undo’ action • Audit Log • Tracking object value changes over time? • Time series?
- 94.
- 95.
- 96. THIRDCHANNEL Agenda • Event Sourcing •Event Sourcing Challenges • Don’t Worry, ES Is Worth It • Implementation (Theory)
- 97. THIRDCHANNEL Implementation (Theory) • SimpleBase Objects
- 98.
- 99.
- 100.
- 101.
- 102.
- 103. THIRDCHANNEL Implementation (Theory) • SimpleBase Objects • Aggregate and Event SubClasses Have Transient Properties
- 104.
- 105.
- 106. THIRDCHANNEL Implementation (Theory) • SimpleBase Objects • Aggregate and Event SubClasses Have Transient Properties • Aggregates Receive and Play Events
- 107.
- 109. THIRDCHANNEL Implementation (Theory) • SimpleBase Objects • Aggregate and Event SubClasses Have Transient Properties • Aggregates Receive and Play Events • Aggregates Require Distinction Between New and Historical Changes
- 110.
- 111. THIRDCHANNEL Implementation (Theory) • SimpleBase Objects • Aggregate and Event SubClasses Have Transient Properties • Aggregates Receive and Play Events • Aggregates Require Distinction Between New and Historical Changes • Event Service Layer Required
- 113. THIRDCHANNEL Agenda • Event Sourcing •Event Sourcing Challenges • Don’t Worry, ES Is Worth It • Implementation (Theory) • Implementation (Reality)
- 114. THIRDCHANNEL Implementation (Reality) • FreelyAdd New Transient Properties
- 115. THIRDCHANNEL Implementation (Reality) • FreelyAdd New Transient Properties • Be Wary of Modifying Events
- 116. THIRDCHANNEL Implementation (Reality) • FreelyAdd New Transient Properties • Be Wary of Modifying Events • Be Wary of Removing Aggregate Transients
- 117. THIRDCHANNEL Implementation (Reality) • FreelyAdd New Transient Properties • Be Wary of Modifying Events • Be Wary of Removing Aggregate Transients • Snapshot Rarely
- 118. Plus so muchmore…
- 119. THIRDCHANNEL Agenda • Event Sourcing •Event Sourcing Challenges • Don’t Worry, ES Is Worth It • Implementation (Theory) • Implementation (Reality) • Persistance
- 120. THIRDCHANNEL Persistence & EventStorage • No Real Need for an ORM
- 121.
- 122. THIRDCHANNEL Persistence & EventStorage • No Real Need for an ORM • Try a Library
- 123. THIRDCHANNEL Event Sourcing Libraries •NEventStore (.NET) • ProophEventStore (PHP) • Akka Persistence (JVM)
- 124. THIRDCHANNEL Persistence & EventStorage • No Real Need for an ORM • Try a Library • Database Agnostic
- 125.
- 126.
- 127.
- 128. THIRDCHANNEL Persistence & EventStorage • No Real Need for an ORM • Database Agnostic • Consider Using a Relational DB
- 129.
- 130. THIRDCHANNEL Agenda • Event Sourcing •Event Sourcing Challenges • Don’t Worry, ES Is Worth It • Implementation (Theory) • Implementation (Reality) • Persistance • Querying
- 132. THIRDCHANNEL Querying Events • AllQueries are Projections, including Current State
- 133. THIRDCHANNEL Querying Events • AllQueries are Projections, including Current State • Returning Current State is Easy
- 134.
- 135. THIRDCHANNEL Querying Events • AllQueries are Projections, including Current State • Returning Current State is Easy • Other Projections Can Be Tough
- 136. Query across EventTypes
- 137.
- 138. THIRDCHANNEL Querying Events • AllQueries are Projections, including Current State • Returning Current State is Easy • Other Projections Can Be Tough • Try Reactive Streams
- 139.
- 140. Use Streams toFilter Events in Code
- 141.
- 142. THIRDCHANNEL Querying Events • AllQueries are Projections, including Current State • Returning Current State is Easy • Other Projections Can Be Tough • Try Reactive Streams • Initial Projection Work May Require Dev Time
- 143. Find Someone witha Statistics Background
- 144. THIRDCHANNEL Querying Events • AllQueries are Projections, including Current State • Returning Current State is Easy • Other Projections Can Be Tough • Try Reactive Streams • Initial Projection Work May Require Dev Time • Consider Cacheable Query Layers
- 145. Separate Event Storage/ Querying Services
- 146.
- 147. THIRDCHANNEL Agenda • Event Sourcing •Event Sourcing Challenges • Don’t Worry, ES Is Worth It • Implementation (Theory) • Implementation (Reality) • Persistance • Querying • Demo Time
- 148. Pray to thedemo gods.
- 149.
- 150.
- 151. Event Sourcing ->Additive Only, Lossless. Can Be Tricky To Work With
- 152. Not for yourMom’s Blog
- 153.
- 154.
- 155. THIRDCHANNEL Image Credits • BOA:http://marketmadhouse.com/wp-content/uploads/2014/11/BofA.jpg • Khaaaaan: http://d1oi7t5trwfj5d.cloudfront.net/38/0d9b80ae5311e1bcc4123138165f92/file/5-things-you-might-not-know-about-star-trek-ii-wrath-of-khan-30th-anniversary.jpg • tribbles: http://de.memory-alpha.org/wiki/Tribble • bank ledger: http://www.iowall.net/accounting-ledger-book.html • ORM: https://guides.codepath.com/android/ActiveAndroid-Guide • hoarders: http://www.aetv.com/hoarders • snapshot stream: https://geteventstore.com/ • office space bobs: http://galleryhip.com/office-space-bobs.html • Nye, Obama, Tyson photo: http://en.wikipedia.org/wiki/Selfie • Printing Press: http://www.directindustry.com/prod/koenig-bauer-ag/offset-printing-presses-high-volume-40462-602574.html
Editor's Notes
- #3 Before I begin, I just want to run through a quick scenario with you all
- #4 Picture your bank in your mind. For many of us, this may not exactly be a happy thought. There’s likely several brick and mortar branches for your particular bank in the area. However, being tech folks, I don’t suppose it’s a stretch to surmise that you all primarily interact with your bank via their website, yes?
- #5 Imagine you went to your bank’s wonderful website, entered your information, and logged in successfully. I hope no one works for farmers. I typed in “Bank website” into google, and this was the first result.
- #6 And, upon logging in, you click the link to check your balance. In doing so, you’re presented with a screen that just shows you “Balance, $100”, but with no context around that number. This may be fine if you expected there to be $100…
- #7 But what if the balance was negative and you weren’t expecting that? What happened to the christmas check from grandma that should have cleared by now?
- #8 What if that’s all your bank balance was, just a simple number? What if that was all your bank could tell you? What if your balance was simply a column in a row in a database somewhere? … and What if you didn’t agree?
- #9 How angry would you be? Can you imagine the arguments you’d have with the teller or an agent over the phone, trying to figure out if your latest pay check was deposited?
- #10 Luckily, that’s not how things are done. Banks store your account’s entire history. Every transaction you make with your bank. Every Credit or Debit made is logged, along with an audit trail of who (e.g. which teller) made the change. To get your balance, your bank simply adds up each of these transactions May also periodically record what the balance was at specific points in time, to prevent having to recalculate everything from the beginning of time.
- #11 There’s a certain advantage to this idea, that we can record all modifications to our data - in this case, the credits and the debits - as EVENTS that occur within our system. For example, Your bank is able to tell you exactly how they arrived at your account balance.
- #12 -What about your company’s software? -Can you tell your users or internal business analysts how you are arriving at the data your application presents to them? … -Today I’m going to present a method called Event Sourcing that does just… that and how it can fuel the competitiveness of your company.
- #13 Bold, right? With that, this is ‘Richer Data History with Event Sourcing’. My name is Steve, and I work for a company called ThirdChannel which is located right across the river in Cambridge.
- #14 Today I’d like to go over the following topics: -Event Sourcing at a high level, -Challenges Add query!
- #15 Let’s begin
- #17 Where an object in memory maps directly to a row in a database, even if that row may be split via joins * update is made to a model, updates a column in your database * in this method, the only thing you know about your data is what it looks like right now.
- #18 Event Sourcing says “that’s fine, but we’re going to something a bit different”. Instead of storing the current state of our models, we’re going to store facts about our models
- #19 Every successful user interaction (and that word successful is important, which I’ll get to later) generates a series of facts or ‘events’ within our system
- #20 This stream of events are persisted in our database in the order they occurred, as a journal of what has transpired. That ’s’ is supposed to be ‘current state’. meant to change this slide
- #21 These events can then be played back against our Domain Object, building it up to the state it would be at any given point in time, although this is most likely the Current State A stream of events represent a particular object in Aggregate
- #22 Which means I should talk about the two main concepts behind Event Sourcing
- #23 An Event Represents something that has occurred within your system. The past tense is important when describing them. It represents an intentional user action or result of user action that almost always results in the manipulation or state change or an Object. Things like “BankAccountOpened”, “CurrencyDeposited” are decent names for events
- #24 The objects which are affected by Events are referred to as an Aggregate. They generally serve as a root for a stream of events; they represent the state of an event stream ‘in aggregate’. Akin to a domain model. It doesn’t have to be, though. It can be, say, relationships between objects. For example, at ThirdChannel we model the assignment relationship between our users and what we call programs as an Aggregate. Along with many other things.
- #25 Event Sourcing is a purely additive model…
- #26 there are no deletes or updating of events. Events are immutable once written to our journal This is a powerful notion, if you consider the implications: using Event Sourcing, no data is ever lost or ignored. <pause>
- #27 Now, When I need to retrieve information about my Aggregates, I simply play back all of the events that have occurred in the past in order to build up the data to a specific point in time, generally the current date, thus getting the current state of our data.
- #28 One of the Key points: by maintaining all events, we’re able to access the current state of our aggregates (again, or objects), certainly, but we can also access the state of our data or aggregates at any point in time. Which is huge.
- #29 I’m sure some of you are thinking “waiiit, if I never get rid of anything, certainly that has tradeoffs, too?” Specifically, performance. What happens if I have thousands, or even millions of events I have to apply?
- #30 You’re right, and that’s a great observation. Luckily, Event Sourcing recommends an early optimization known as ‘Snapshots’
- #31 A Snapshot is just what you’d think it would be: a recording of the details of your Aggregate at that moment in time. Persisted forever
- #32 As we consume and create events, periodically we persist a snapshot, containing the state of the aggregate at that point in time. -When replaying events, we load from the most recent snapshot, then apply only the events between when that snapshot was taken, and the targeted end date. So, in this case,… -I’ll get into some more specifics around snapshots in a bit.
- #33 Before I get to the next section, I’m going to go over another example that I’ll reference back to periodically
- #34 Suppose we were building an ecommerce app and we are building the ‘shopping’ cart feature. This is one of my favorite examples, by the way.
- #35 Naive, ORM, relational -> join table with quantity Event Sourcing -> <identify page components> are not saved as a join table or a single row.
- #36 Instead, w/ ES, system stores all commands you’ve issued / replays for current state <list events> Quickly remove base from the cart, before placing the order and generating an OrderPlacedEvent View doesn’t display raw events Data backing the view is built up from events to form an Object intended for View Object is Transient -> object will be garbage collected and no direct representation exists on disk
- #37 That brings up the next step, Working with Objects
- #38 And this is where Event Sourcing will start to hurt your brain. In order to fully grasp what Event Sourcing is, it’s important to realize that…:
- #39 All objects that are ‘displayed’ to the user in your View layer are simply transient derivatives of your event stream. They are ephemeral and must be built up from the events to be used in a traditional manner within your application
- #40 Finally, I argue that structuring our data in this way is akin to the way our brains work; it’s natural.
- #41 Internally, your mind is able to tell you the current state of your knowledge about things. This current state is formed by a series of observations / facts / events in your past. You’re able to replay these events in your mind, and also remember your knowledge at that point in time. Our minds aren’t perfect, though, and sometimes we violate the ES rules by deleting Events. Ooops.
- #42 Let’s take me as an Example. Even if you’ve never seen or met me before today, your mind has already recorded a series of facts which is driving your mental model of me. For example: FeatureObservedEvent ActionObservedEvent -Now, if I were to suddenly make a rude gesture at the audience…
- #43 that would apply a new event to your mental model. Your current state opinion of me would likely be negative although you could remember a time before you thought negatively of me. “Man, Steve seemed like an alright guy until he flipped off the audience. What a jerk”
- #44 That, I think, is the basis for what Event Sourcing is. It may be a bit early, but are there any questions so far?
- #45 Then Let’s move on to the next section, Challenges or Difficulties with ES. Or as I like to call it…
- #47 Right now, you may be suspicious. You may be thinking:
- #48 “I mean, what you’re describing sounds like a ton of extra work to implement.”
- #49 -Not to mention a ton of overhead in processing these events, even if we do make good use of these snapshots you mention
- #50 How can you operate in a world without Models?!
- #51 And yes, that’s true. *pause* Furthermore, here’s some more bad things:
- #52 Storing every event that occurs within your system will almost certainly require more storage space
- #53 hoarders
- #54 Now this is a truly difficult one. - Will have reduced Database Level Constraints, like Foreign Keys, null checks, unique checks, etc. - Instead, we have to rely on our software for transactions and these database constraints
- #55 Now, this is usually where I lose the more seasoned developers. Because our properties tend to be transients, serialized within the event, we lose things like foreign key constraints at the db level Instead, in Event Sourcing, these checks tend to move within Transactional blocks within your code. Unsure what to put for this slide. I’ve been using a good deal of Spring and Hibernate lately, but thought if I just put a @Transactional annotation, that might be strange.
- #56 Plus, ES can also be difficult for Junior Engineers I’ve noticed that people really cling to the Model View Controller way of life. - This is a very different way of building our applications, particularly for the web Recommending a different structure for the Model can make people wary. - Telling people that the Model becomes a “Transient object derived from the event stream” scares them
- #57 About a year ago, I was talking with a group at a Meetup about this very subject. Afterwards I was confronted by a rather ornery fellow. He said, and I paraphrase: “Steve, I can see you are passionate about this subject… but I gotta tell you that it’s just really, really dumb….”
- #58 -I have access to tons of log files - If I wanted all of my events, I’d just search / harvest my database’s transaction log! That has everything!” - He was a lot angrier and confrontational than a normal human should be. I withered under his furious comments.
- #59 -But, his comment about the transaction log is interesting. Does everyone know about this? -Relational databases (and Redis I know has a variation of this) will save every action taken against your data, and in the event of error recovery, can use the log to rebuild the current state of your database. - A transaction log is actually a series of Events used … to recreate the current state of your data. Sound familiar?
- #60 As crazy as this all might sound, I’d like to argue that Event Sourcing actually has huge Benefits
- #61 Next Up, “don’t worry, ES is worth it”
- #62 First, Going back to the concept of a transaction or an Audit Log… Why is it that I’d want something fancy like ES, when I can audit a log file or look at my transaction log?
- #63 There’s a subtle difference between an Audit Log and an Event Stream. Audit logs tell the history of what happened in your system or what was persisted to the database. Events Tell
- #64 Furthermore, Having the Events as a first-order member of your platform can give you enhanced information around what your users or systems are doing beyond what gets written to the database. And it’s easier to work with if the events are integrated within the platform already.
- #65 Incidentally, an event object typically will have attached to it information about the user which generated the event, which also makes ES a perfect Audit Log
- #66 Data storage is crazy cheap. Last I looked, AWS basic SSD storage was 0.013 per gigabyte hour/ which equates to… If you’re at the point where those pennies matter, you probably have bigger problems.
- #67 next. What I find interesting is that Event Sourcing, or a non-digital analogue of it, is used by every ‘mature’, or long running business.
- #68 Just Like I went over in the beginning, banks and accounting methods operate with Event Sourcing Bankers additionally even use snapshots of your balance in an additional column beyond credits and debits
- #69 Lawyers! If a contract needs to be adjusted, is the contract thrown out and re-written? No. Rather, ‘addendums’ are placed on the contract. To figure out the contract actually involves, one has to read the initial contract, and then each successful addendum in order to figure out what the thing actually says.
- #70 In addition, I argue that all business problems can be successfully modeled with - and benefit greatly by - event sourcing
- #71 How many of you all have delete statements in your code?
- #72 Remember: there is no delete, ES is the only structural model which does not lose information.
- #73 Event Sourcing simplifies Testing and Debugging. A bold claim, I know.
- #74 Testing is easier / simplified with ES, you unit test the events, then later you can simply assert that specific events are applied during integration testing
- #75 In addition, debugging is easy, because we have the entire history of our data.
- #76 We can look back through our Aggregates’ timelines <next> and examine them at any point in history.
- #77 Thus I can see what the historical state of the aggregate… or all my aggregates… was at a particular point in time, along with how it reached that state and who caused those changes.
- #78 ahem… I’m sure you’re all keenly aware, but 2015 is the year they visited in this movie. “Where’s my Hoverboard?!” Anyway. If we at some point note that there’s an error or discrepancy in our data…
- #79 debugging or tracing the error is a snap -We can find the faulty or conflicting event, know who executed it, when they executed it, and what lead up to the bad state. And then we can emit a new event to ‘patch’ the issue
- #80 If we want to get even crazier, we could go to a specific point in our data’s timeline, then fire fake events in order to simulate alternate timelines.
- #81 pause This has interesting applications for, say, a.b. testing, stress, and disaster testing. If any of these past few slides reminds you of git… well, how astute. Git is like recursive event sourcing. Ever look at the reflog?
- #82 Event sourcing is the ideal storage mechanism for business analysis
- #83 -because Event Sourced systems do not lose data, they’re future proofed against any crazy reports that your business analysts may need in the future
- #84 suppose one such analyst came to your Ecommerce / shopping cart team asking for… all shoppers who add items to their cart and then remove them within 5 minutes. They want to know who, and which products
- #85 with non es, and the naive way I mentioned earlier, you might have to build some sort of tracking table, or mark additional rows with a timestamp.. I dunno. Regardless, then you deploy… and then wait for the data to gather, as users add and remove products
- #86 with Event Sourcing, your write a query for that report, you deploy… and then what to do you have? If you’re thinking: all of the data, obviously… well, you’re wrong. You have MORE than everything. We can generate the report for how it would look at every point in our history.
- #87 Which makes the company and your business analysts extremely happy. There’s nothing they like better than a good report.
- #88 Querying over the events; presenting different Views on them, is often called a projection Perhaps the biggest advantage of ES for me. look at specific events across one stream look at specific event types across all streams
- #89 Grabbing the Current State of an aggregate is a projection, and probably the easiest: take all events for an Aggregate, in order. There’s a good deal else to find. In our shopping cart example: find items in cart for any given date or time range find items that were removed for any given date or time range find average rate of items removed vs items purchased for any given… find average duration between items being added and then being removed for any…
- #91 All of the current FMRs or Agents (excuse the font, please!) which one of us made those transitions a timeline of each agent’s transition within the program applications with a long gap between entering the system and being wait listed or interviewed, to see how long a candidate waits until we contact them how long on average, an agent lasts before being fired and/or average time for agents that have quit Turnover for a date range And I can tell you that information at ANY POINT IN TIME. e.g. the average quit rate might be different now than 6 months ago, for example that’s all I could think of off the top of my head Which is amazing, right? Just from that one relation. What would happen.. if I started to correlate other event streams?
- #92 <Pause> Even after this presentation, if you’re still skeptical of the benefits… and you think this is the silliest thing you’ve ever heard of
- #93 ..be aware that the decision can be out of your hands. Event Sourcing is often chosen or driven by Management out of business needs, and ‘hacky’ analogues are shoehorned into an existing system AFTER the fact. Rather, no one says “We need Event Sourcing”, but there’s discussions around the problems that need solving
- #94 I know I have
- #96 Anyway, next up is implementation
- #97 first let’s discuss the theoretical approaches.
- #98 Pure Event Sourcing is fairly simplistic in terms of implementation
- #99 There’s really only 3 base objects that you have to worry about.
- #100 -First up, the Aggregate. You have the id, which should be a UUID, the current revision number, and the type (or ‘clazz’ if you’re working with java) -The current revision number is used for optimistic locking and to see how advanced our aggregate is. -The type is used by our system when we want to load the aggregate into a more meaningful class in the system, say, a SubClass of Aggregate. In our example, the ShoppingCart class would SubClass from Aggregate
- #101 Next up, Event -id, revision aggregate_id, the date with time stamp, the type, the user id, and then ‘data’ -data is a serialized representation of the event type’s properties. Generally, JSON, XML, or are recommended for the storage mechanism in the data column. - this could also be a more efficient mechanism, too, like Google’s Protocol Buffers or Apache Avro
- #102 All Events should be named in the past tense, as they should reflect something successful that happened in the PAST
- #103 Lastly, we have Snapshot. Again, we want to serialize the properties of the aggregate at that moment in time. Next,
- #104 I should be clear about what - exactly - is being serialized into those data fields. Aggregates and Events, or at least classes that implement Aggregate and Event, contain, themselves, transient properties which are generally not persisted to the database.
- #105 Plain old object with explicit transient properties each has corresponding event or events
- #106 The Event itself has transient properties, whose values are persisted to the database. Also, if anyone notices that I’m using JPA annotations and I earlier mentioned that this is an alternative to ORMs… appreciate the irony. This is from a small demo app.
- #108 Events Modify Transient values on the Aggregate
- #109 It’s almost a Visitor pattern. As Events are generated, they are applied to an Aggregate. Aggregates are built up or, in the case of loading an aggregate, building back up. Event by Event my actual aggregate class may have several properties, however, they are all transient, in the sense that they are not persisted local to the aggregate… e.g. not in the same table. When the aggregate is first created, all of these transients are at their default value, and the playback of the events will restore them to whichever point in time I want.
- #111 shopping cart -> Order placement should only charge credit card the first time the event is created
- #112 In addition, you’ll also need a service layer to store and load the events and aggregates
- #113 it must remember to load events in order of their revision number for the correct aggregate, and initiate the event serialization and de-serialization processes
- #114 And those are the basics. That’s not too bad, eh? Unfortunately… there are a few more practical considerations to go over that are a reality for any real ES system.
- #115 When working with ES, feel free to add new Transient properties to aggregates and Events. Typically, one adds a new transient property to their Aggregate, and then have one or more corresponding new events. Incidentally, no database changes will be required.
- #116 event ‘versioning’ when hydrating, have to know how to deal with old event versions and new events
- #117 Because an aggregate has a set of transient properties that are manipulated by events, removing one can cause havoc on the events that were there.
- #118 While snapshots seem awesome, do it only rarely. Greg Young, one of the largest voices in the ES community, claims that he doesn’t snapshot an aggregate until it reaches 1000 events. You have to juggle the time cost of the additional query for the snapshot versus the processing of the small event objects.
- #119 I had originally built this out to enumerate many more items, but then realized that they could be grouped into distinct sections. So, next up…
- #120 Persistance: How, exactly, do we store these events?
- #122 With the naive use case, here’s my entire schema… our at least, without Snapshot. The snapshot is very similar to aggregate, just with a data text field
- #123 When looking for a storage mechanism, there many options available to you, and, for the most part, you’ll be fine no matter which database option you choose. Now, here are some of the better options
- #124 These are the ones I’m aware of that are in active development. NVentStore, Prooph, and Akka. Akka is interesting, in that every object you work with is persisted as an event stream, but it doesn’t explicitly call itself Akka Event Source. Of these, I’d probably recommend Akka, provided you’re on the JVM. The persistence component is available as a standalone jar
- #125 When looking for a storage mechanism, there many options available to you, and, for the most part, you’ll be fine no matter which database option you choose. Now, here are some of the better options
- #126 First, EventStore, the database. Highly specialized for working with events and generating projections. Written By Greg Young, who is perhaps the most well known person in the Event Sourcing community.
- #127 Redis is a fantastic choice. It’s excellent for a rapid writes. Furthermore, Redis itself has a few options for persisting to disk, one of which is event sourcing.
- #128 And if course, you cannot go wrong with good, old fashioned Relational Databases.
- #129 I would suggest consider sticking with a standard relation database, if you’re already using it Switching to something like Event Sourcing is already enough change
- #131 And now we get to a particular hairy topic…
- #132 How am I going to handle all of these events and find what I’m looking for?
- #133 All queries within ES are often referred to as a Projections over the Event Stream. This includes the concept of the current state.
- #134 All queries within ES are often referred to as a Projection over the Event Stream. This includes the concept of the current state.
- #135 Load the aggregate or aggregates by id, load all of the events and replay the events to get back to current state Fairly easy and straightforward
- #136 All queries within ES are often referred to as a Projection over the Event Stream. This includes the concept of the current state.
- #137 If we’re querying across the current state of the properties, we can: query the most recent of the events responsible for that property. User Role Or.. Had a rather onerous query from our codebase, but then decided a few minutes ago that it’s probably not a great idea to show proprietary / embarrassing code.
- #138 maintain a synchronized copy of your aggregates in another table; always sync current state into it, and then query that table I generally like to keep any data synchronization to a minimum, but this approach can be an attractive convenience for current state searches
- #140 Has everyone heard of stream programming, reactive streams, or reactive extensions? I’m primarily used to the JVM space, so coming from there, we have the ReactiveX libraries (which comes out of netflix, bindings for many languages), Reactor, Java 8’s stream api, and I’m sure the people at TypeSafe could something to this. This is a topic that can be a presentation unto itself
- #142 In fact, the Event Store database, that’s exactly what they do. One has to write Projections using Streams within a web interface, that then become query-able by clients.
- #143 However, as you can probably see. This is fairly difficult. The development team will need to spend time writing the projections. If you have analysts on your team that are used to writing sql, well, it’s going to be much more difficult for them.
- #144 In fact, you may want to hire a fabled Data Scientists, or at least find someone who has experience with Statistics. I believe that they would have the best chance of finding meaning in the swaths of historical data you’re gathering.
- #145 Because this can be so tricky to work with, and because I generally don’t like synchronization, I would absolutely recommend building a query able layer in cache.
- #146 In addition, one thing we do at 3c for some of our data is to have a separate service which pulls data from the service gathering the events, and packages it up nicely for querying by other services. separate reporting / read service
- #147 Consider feeding events into additional services or tools, particularly those that are stream friendly, like Splunk or Apache Storm
- #150 First, I’d like to point out that we at ThirdChannel have open sourced a small library that we’re using internally for doing Event Sourcing on the JVM. took the screenshot as unsure of the capacity of the wifi here
- #152 Event Sourcing is an additive only, lossless data storage pattern which has insanely high potential for data analysis. It is, however, tricky to work efficiently with.
- #153 I wouldn’t recommend it for certain applications; say small static content information websites (e.g. a restaurant or a business’ marketing site). Nor does it make sense to apply to every domain object in a system. However, key data in your application that you use to drive your business can benefit greatly from this approach.
- #154 With that, I say thank you for you coming.