Append only data stores

5,757 views

Published on

Malmö breakfast seminar 2013-10-16

Published in: Entertainment & Humor, Design
0 Comments
4 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
5,757
On SlideShare
0
From Embeds
0
Number of Embeds
26
Actions
Shares
0
Downloads
29
Comments
0
Likes
4
Embeds 0
No embeds

No notes for slide

Append only data stores

  1. 1. Append-only data stores vs onsdag 16 oktober 13
  2. 2. What? onsdag 16 oktober 13
  3. 3. Append-only data stores Only add, never remove or change Can retrieve old values onsdag 16 oktober 13
  4. 4. Source code Version controlled Keep all versions onsdag 16 oktober 13
  5. 5. The problem onsdag 16 oktober 13
  6. 6. Mutable state id email jakr onsdag 16 oktober 13 name Jan Kronquist jan.kronquist@jayway.se
  7. 7. Mutable state id name email jakr Jan Kronquist jan.kronquist@jayway.se UPDATE customer SET email="jan.kronquist@jayway.com" WHERE id="jakr" onsdag 16 oktober 13
  8. 8. Mutable state id name email jakr Jan Kronquist jan.kronquist@jayway.com UPDATE customer SET email="jan.kronquist@jayway.com" WHERE id="jakr" onsdag 16 oktober 13
  9. 9. Mutable state id name email jakr Jan Kronquist jan.kronquist@jayway.com UPDATE customer SET email="jan.kronquist@jayway.com" WHERE id="jakr" I didn’t get confirmation of the order last week? onsdag 16 oktober 13
  10. 10. Mutable state id name email jakr Jan Kronquist jan.kronquist@jayway.com UPDATE customer SET email="jan.kronquist@jayway.com" WHERE id="jakr" I didn’t get confirmation of the order last week? Could you reset my email address? onsdag 16 oktober 13
  11. 11. The technical details SELECT count(*) AS inSweden FROM customer WHERE email LIKE "%@jayway.se" SELECT count(*) AS elsewhere FROM customer WHERE email LIKE "%@jayway.com" total = inSweden + elsewhere onsdag 16 oktober 13
  12. 12. Human errors DELETE FROM customer WHERE email LIKE "j%" onsdag 16 oktober 13
  13. 13. Scalability - the easy parts Caching Static content onsdag 16 oktober 13
  14. 14. Scalability - the easy parts Caching Static content onsdag 16 oktober 13
  15. 15. Scalability - the easy parts Caching Static content Stateless work onsdag 16 oktober 13
  16. 16. Scalability - the easy parts Caching Static content Stateless work Load balancer onsdag 16 oktober 13
  17. 17. Scalability - the hard parts State + behavior onsdag 16 oktober 13
  18. 18. Scalability - the hard parts State + behavior onsdag 16 oktober 13
  19. 19. Example domain onsdag 16 oktober 13
  20. 20. Rock - Paper - Scissors onsdag 16 oktober 13
  21. 21. http://rock-paper-scissors.com/ The future Facebook of Rock Paper Scissors Millions of users Many games per user onsdag 16 oktober 13
  22. 22. Playing the game Player A Player B onsdag 16 oktober 13 Server
  23. 23. Playing the game Player A Server Game 123 Player B onsdag 16 oktober 13
  24. 24. Playing the game Player A paper Server Game 123 Player B onsdag 16 oktober 13
  25. 25. Playing the game Player A paper Server Game 123 Player B: paper Player B onsdag 16 oktober 13
  26. 26. Playing the game rock Player A Server Game 123 Player B: paper Player A: rock Player B onsdag 16 oktober 13
  27. 27. Playing the game Player A Game 123 winner: Player B loser: Player A Server Game 123 Player B: paper Player A: rock Player B onsdag 16 oktober 13
  28. 28. Playing the game GAME WON other move (victory) CREATED WAITING any move other move (tie) GAME TIED Player A Game 123 winner: Player B loser: Player A Server Game 123 Player B: paper Player A: rock Player B onsdag 16 oktober 13
  29. 29. Entities onsdag 16 oktober 13
  30. 30. Entities Player + id + name + email onsdag 16 oktober 13
  31. 31. Entities + + + + + + onsdag 16 oktober 13 Game id state players moves winner loser Player + id + name + email
  32. 32. Entities + + + + + + onsdag 16 oktober 13 Game id state players moves winner loser Player + id + name + email Move + move + player
  33. 33. Actions Create game POST /games Make move POST /games/123 move=rock Get game result GET /games/123 onsdag 16 oktober 13
  34. 34. Actions Create game POST /games Make move POST /games/123 move=rock Get game result GET /games/123 1. Form (make move) 2. Waiting 3. Game result 4. Error onsdag 16 oktober 13
  35. 35. Clojure onsdag 16 oktober 13
  36. 36. Clojure data structures onsdag 16 oktober 13
  37. 37. Clojure data structures Lists - singly linked (1 2 3 4 5) (fred ethel lucy) (list 1 2 3) onsdag 16 oktober 13
  38. 38. Clojure data structures Lists - singly linked (1 2 3 4 5) (fred ethel lucy) (list 1 2 3) Vectors - indexed access [1 2 3] onsdag 16 oktober 13 ["fred" 17 3.14 "bar"]
  39. 39. Clojure data structures Lists - singly linked (1 2 3 4 5) (fred ethel lucy) (list 1 2 3) Vectors - indexed access [1 2 3] ["fred" 17 3.14 "bar"] Maps {:name onsdag 16 oktober 13 "Jan Kronquist" :age 37}
  40. 40. Event Sourcing with Event Store onsdag 16 oktober 13
  41. 41. Event Sourcing Events that have happened Ordered in time Source of truth onsdag 16 oktober 13
  42. 42. http://geteventstore.com Runs on .NET and Mono Free (store) or Commercial (HA) API €1500/year Custom TCP (c#) ATOM over HTTP (xml/json) Features Event streams, projections, generate new events (CEP) onsdag 16 oktober 13
  43. 43. Event example Server onsdag 16 oktober 13 EventStore
  44. 44. Event example CreateGameCommand{:game 123 :creator "player-1"} Server onsdag 16 oktober 13 EventStore
  45. 45. Event example CreateGameCommand{:game 123 :creator "player-1"} Server EventStore GameCreatedEvent{:game 123 :creator "player-1"} onsdag 16 oktober 13
  46. 46. Event example MakeMoveCommand{:game 123 :player "player-1" :move "rock"} Server EventStore GameCreatedEvent{:game 123 :creator "player-1"} MoveMadeEvent{:game 123 :player "player-1" :move "rock"} onsdag 16 oktober 13
  47. 47. Event example MakeMoveCommand{:game 123 :player "player-2" :move "paper"} Server EventStore GameCreatedEvent{:game 123 :creator "player-1"} MoveMadeEvent{:game 123 :player "player-1" :move "rock"} MoveMadeEvent{:game 123 :player "player-2" :move "paper"} GameWonEvent{:game 123 :winner "player-2" :loser "player-1"} onsdag 16 oktober 13
  48. 48. Current state? GameCreatedEvent{:game 123, :creator "player-1", :time 1} MoveMadeEvent{:game 123, :player "player-1", :move "rock", :time 2} onsdag 16 oktober 13
  49. 49. Current state? GameCreatedEvent{:game 123, :creator "player-1", :time 1} MoveMadeEvent{:game 123, :player "player-1", :move "rock", :time 2} GAME WON if victory CREATED WAITING any move other move if tie GAME TIED onsdag 16 oktober 13
  50. 50. Current state? GameCreatedEvent GameCreatedEvent{:game 123, :creator "player-1", :time 1} MoveMadeEvent{:game 123, :player "player-1", :move "rock", :time 2} GAME WON if victory CREATED WAITING any move other move if tie GAME TIED onsdag 16 oktober 13
  51. 51. Current state? GameCreatedEvent{:game 123, :creator "player-1", :time 1} MoveMadeEvent MoveMadeEvent{:game 123, :player "player-1", :move "rock", :time 2} GAME WON if victory CREATED WAITING any move other move if tie GAME TIED onsdag 16 oktober 13
  52. 52. Aggregates Scope of consistency when handling a command Game + id + state + players + moves + winner + loser onsdag 16 oktober 13 Move Player + id + id + move + name + player + email
  53. 53. Aggregates as Event Streams stream = game-1 version = 2 onsdag 16 oktober 13 event1 GameCreatedEvent{:game 1...} event2 MoveMadeEvent{:game 1....}
  54. 54. Aggregates as Event Streams stream = game-1 version = 2 event1 GameCreatedEvent{:game 1...} event2 MoveMadeEvent{:game 1....} stream = game-2 version = 4 event1 event2 event3 event4 onsdag 16 oktober 13 GameCreatedEvent{:game 2...} MoveMadeEvent{:game 2....} MoveMadeEvent{:game 2....} GameWonEvent{:game 2....}
  55. 55. Aggregates as Event Streams stream = game-1 version = 2 event1 GameCreatedEvent{:game 1...} event2 MoveMadeEvent{:game 1....} stream = game-2 version = 4 event1 event2 event3 event4 GameCreatedEvent{:game 2...} MoveMadeEvent{:game 2....} MoveMadeEvent{:game 2....} GameWonEvent{:game 2....} curl "http://127.0.0.1:2113/streams/game-1" -d @eventdata.json -H "Content-Type:application/json" -H "ES-ExpectedVersion: 2" onsdag 16 oktober 13
  56. 56. Projections GameCreatedEvent{:game 1...} MoveMadeEvent{:game 1....} GameCreatedEvent{:game 2...} MoveMadeEvent{:game 2....} MoveMadeEvent{:game 2....} GameWonEvent{:game 2....} onsdag 16 oktober 13
  57. 57. Projections GameCreatedEvent{:game 1...} MoveMadeEvent{:game 1....} GameCreatedEvent{:game 2...} MoveMadeEvent{:game 2....} MoveMadeEvent{:game 2....} GameWonEvent{:game 2....} for (Event events : allEvents) { if (event instanceof GameCreatedEvent) { created++; } } return created; onsdag 16 oktober 13
  58. 58. Projections GameCreatedEvent{:game 1...} MoveMadeEvent{:game 1....} GameCreatedEvent{:game 2...} MoveMadeEvent{:game 2....} MoveMadeEvent{:game 2....} GameWonEvent{:game 2....} onsdag 16 oktober 13
  59. 59. Projections GameCreatedEvent{:game 1...} MoveMadeEvent{:game 1....} GameCreatedEvent{:game 2...} MoveMadeEvent{:game 2....} MoveMadeEvent{:game 2....} GameWonEvent{:game 2....} onsdag 16 oktober 13 created = 2
  60. 60. Projections GameCreatedEvent{:game 1...} MoveMadeEvent{:game 1....} GameCreatedEvent{:game 2...} MoveMadeEvent{:game 2....} MoveMadeEvent{:game 2....} GameWonEvent{:game 2....} created = 2 GameCreatedEvent{:game 3...} onsdag 16 oktober 13
  61. 61. Projections GameCreatedEvent{:game 1...} MoveMadeEvent{:game 1....} GameCreatedEvent{:game 2...} MoveMadeEvent{:game 2....} MoveMadeEvent{:game 2....} GameWonEvent{:game 2....} created = 2 GameCreatedEvent{:game 3...} (state, event) -> { if (event instanceof GameCreatedEvent) { return state++; } rerturn state; } onsdag 16 oktober 13
  62. 62. Projections in Javascript fromAll() .when({ $init: function () { return { created: 0, completed:0 }; }, }); onsdag 16 oktober 13
  63. 63. Projections in Javascript fromAll() .when({ $init: function () { return { created: 0, completed:0 }; }, GameCreatedEvent: function(s, e) { s.created++; return s; }, }); onsdag 16 oktober 13
  64. 64. Projections in Javascript fromAll() .when({ $init: function () { return { created: 0, completed:0 }; }, GameCreatedEvent: function(s, e) { s.created++; return s; }, GameWonEvent: function(s, e) { s.completed++; return s; }, }); onsdag 16 oktober 13
  65. 65. Projections in Javascript fromAll() .when({ $init: function () { return { created: 0, completed:0 }; }, GameCreatedEvent: function(s, e) { s.created++; return s; }, GameWonEvent: function(s, e) { s.completed++; return s; }, GameTiedEvent: function(s, e) { s.completed++; return s; } }); onsdag 16 oktober 13
  66. 66. Projections in Javascript fromAll() .when({ $init: function () { return { created: 0, completed:0 }; }, GameCreatedEvent: function(s, e) { s.created++; return s; }, }); onsdag 16 oktober 13 GameWonEvent: function(s, e) { s.completed++; return s; }, /projections/games-counter GameTiedEvent: function(s, e) { { s.completed++; created: 15, return s; completed: 10 } }
  67. 67. External Projections Business logic events onsdag 16 oktober 13
  68. 68. External Projections Business logic Projections events onsdag 16 oktober 13 events
  69. 69. External Projections Business logic Projections events events Available Consistent onsdag 16 oktober 13
  70. 70. External Projections Commands Queries events events Available Consistent onsdag 16 oktober 13
  71. 71. System with EventStore onsdag 16 oktober 13
  72. 72. System with EventStore Command Handler onsdag 16 oktober 13
  73. 73. System with EventStore 1.command onsdag 16 oktober 13 Command Handler
  74. 74. System with EventStore 1.command onsdag 16 oktober 13 Command Handler 2.get events Event Store
  75. 75. System with EventStore 1.command onsdag 16 oktober 13 Command Handler 2.get events 3.add events Event Store
  76. 76. System with EventStore 1.command onsdag 16 oktober 13 Command Handler 2.get events 3.add events 4.update projections Event Store
  77. 77. System with EventStore 1.command Command Handler 2.get events 3.add events 5.get events or projections Some service onsdag 16 oktober 13 4.update projections Event Store
  78. 78. System with EventStore 1.command Command Handler 2.get events 3.add events 4.update projections Event Store 5.get events or projections Some service onsdag 16 oktober 13 6.update state
  79. 79. System with EventStore 1.command Command Handler 2.get events 3.add events 4.update projections Event Store 5.get events or projections 7.query onsdag 16 oktober 13 Some service 6.update state
  80. 80. More info Functional Programming with DDD http://skillsmatter.com/podcast/design-architecture/ddd-functional-programming A deep look into the Event Store http://vimeo.com/53153270 onsdag 16 oktober 13
  81. 81. Datomic onsdag 16 oktober 13
  82. 82. http://www.datomic.com/ Runs on JVM €2300 Free (3 peers, local storage) or Commercial (HA, distrib) Features ACID transactions, declarative query engine onsdag 16 oktober 13
  83. 83. Fact Something known to have happened or existed onsdag 16 oktober 13
  84. 84. Fact Something known to have happened or existed 2004-10-01 the email of Jan was ”jan.kronquist@jayway.se” onsdag 16 oktober 13
  85. 85. Fact Something known to have happened or existed 2004-10-01 the email of Jan was ”jan.kronquist@jayway.se” 2007-01-01 the email of Jan was ”jan.kronquist@jayway.com” onsdag 16 oktober 13
  86. 86. Atomic fact (datom) Entity Attribute Value Time onsdag 16 oktober 13
  87. 87. Atomic fact (datom) Entity The person Jan Kronquist Attribute email Value ”jan.kronquist@jayway.se” Time 2004-10-01 onsdag 16 oktober 13
  88. 88. Data perception Facts Add facts Add more facts Even more facts time onsdag 16 oktober 13
  89. 89. Data perception Facts Add facts Add more facts Even more facts time onsdag 16 oktober 13
  90. 90. Data perception Facts query Add facts Add more facts Even more facts time onsdag 16 oktober 13
  91. 91. Data perception Facts query Add facts Add more facts query Even more facts time onsdag 16 oktober 13
  92. 92. Data perception Facts Add facts Add more facts query Even more facts time onsdag 16 oktober 13
  93. 93. Data perception Facts Add facts Add more facts query Even more facts time onsdag 16 oktober 13
  94. 94. Data perception Facts Add facts Add more facts query Even more facts time onsdag 16 oktober 13
  95. 95. Database deconstructed onsdag 16 oktober 13
  96. 96. Database deconstructed Transactor Consistency by serializing transactions onsdag 16 oktober 13
  97. 97. Database deconstructed Transactor Consistency by serializing transactions Peer Your code + Query engine onsdag 16 oktober 13
  98. 98. Database deconstructed Transactor Consistency by serializing transactions Peer Your code + Query engine Storage service Key/value store Scalable reads and writes eg DynamoDB, InfiniSpan, Riak + ZooKeeper, SQL(!) onsdag 16 oktober 13
  99. 99. System with Datomic Peer A Transactor Distributed onsdag 16 oktober 13 Storage Service
  100. 100. System with Datomic 1.command Peer A Transactor Distributed onsdag 16 oktober 13 Storage Service
  101. 101. System with Datomic 1.command Peer A 2.read and cache Transactor Distributed onsdag 16 oktober 13 Storage Service
  102. 102. System with Datomic 1.command Peer A 3.transact 2.read and cache Transactor Distributed onsdag 16 oktober 13 Storage Service
  103. 103. System with Datomic 1.command Peer A 3.transact 2.read and cache Transactor 4.write Distributed onsdag 16 oktober 13 Storage Service
  104. 104. System with Datomic 1.command Peer A 3.transact 2.read and cache Transactor 4.write 5.notify Peer B onsdag 16 oktober 13 Distributed Storage Service
  105. 105. Datomic example Peer onsdag 16 oktober 13 Transactor
  106. 106. Datomic example CreateGameCommand{:game 123 :creator "player-1"} Peer onsdag 16 oktober 13 Transactor
  107. 107. Datomic example CreateGameCommand{:game 123 :creator "player-1"} Transactor Peer [123 [123 onsdag 16 oktober 13 :creator :state "player-1" :created T1] T1]
  108. 108. Datomic example CreateGameCommand{:game 123 :creator "player-1"} Transactor Peer [123 [123 :creator :state "player-1" :created ["player-1" :email "player@gmail.com" T0] onsdag 16 oktober 13 T1] T1]
  109. 109. Datomic example CreateGameCommand{:game 123 :creator "player-1"} Transactor Peer [123 [123 onsdag 16 oktober 13 :creator :state "player-1" :created T1] T1]
  110. 110. Datomic example CreateGameCommand{:game 123 :creator "player-1"} Transactor Peer [123 [123 :creator :state "player-1" :created T1] T1] GameCreatedEvent{:game 123 :creator "player-1"} onsdag 16 oktober 13
  111. 111. Datomic example CreateGameCommand{:game 123 :creator "player-1"} Transactor Peer [123 [123 onsdag 16 oktober 13 :creator :state "player-1" :created T1] T1]
  112. 112. Datomic example MakeMoveCommand{:game 123 :player "player-1" :move :rock} Transactor Peer [123 [123 [123_1 [123_1 [123 [123 onsdag 16 oktober 13 :creator :state :move :player :moves :state "player-1" :created :rock "player-1" 123_1 :waiting T1] T1] T2] T2] T2] T2]
  113. 113. Datomic example MakeMoveCommand{:game 123 :player "player-2" :move :paper} Transactor Peer [123 [123 [123_1 [123_1 [123 [123 [123_2 [123_2 [123 [123 [123 [123 onsdag 16 oktober 13 :creator :state :move :player :moves :state :move :player :moves :state :winner :loser "player-1" :created :rock "player-1" 123_1 :waiting :paper "player-2" 123_2 :won "player-2" "player-1" T1] T1] T2] T2] T2] T2] T3] T3] T3] T3] T3] T3]
  114. 114. Aggregates? Only by convention cas (compare and set) isComponent on relation Game + id + state + players + moves + winner + loser onsdag 16 oktober 13 Move + id + move + player
  115. 115. Query language What to find Where clauses [entity attribute value] Implicit joins Call arbitrary function onsdag 16 oktober 13
  116. 116. Query example - basic [:find ?email :where [_ :email ?email]] onsdag 16 oktober 13
  117. 117. Query example - basic [:find ?email :where [_ :email ?email]] [["jan.kronquist@jayway.com] ["player-1@gmail.com"] ["player-2@gmail.com"]] onsdag 16 oktober 13
  118. 118. Query example - aggregation [:find ?state (?count game) :where [?game :state ?state]] onsdag 16 oktober 13
  119. 119. Query example - aggregation [:find ?state (?count game) :where [?game :state ?state]] [[:created [:waiting [:won [:tied onsdag 16 oktober 13 4] 1] 7] 3]]
  120. 120. Query example - aggregation [:find ?state (?count game) :where [?game :state ?state]] [[:created [:waiting [:won [:tied 4] 1] 7] 3]] inProgess = created + waiting onsdag 16 oktober 13
  121. 121. Query example - aggregation [:find ?state (?count game) :where [?game :state ?state]] fromAll() .when({ $init: function () { return { created: 0, completed:0 }; }, GameCreatedEvent: function(s, e) { s.created++; return s; }, GameWonEvent: function(s, e) { s.completed++; return s; }, GameTiedEvent: function(s, e) { /projections/games-counter s.completed++; { return s; created: 15, } completed: 10 }); } onsdag 16 oktober 13 [[:created [:waiting [:won [:tied 4] 1] 7] 3]] inProgess = created + waiting
  122. 122. Query example - join [:find ?email (count ?game) :where [?player :email ?email] [?game :winner ?player]] onsdag 16 oktober 13
  123. 123. Query at any time (q ’[:find ?email :where [_ :email ?email]] db) onsdag 16 oktober 13
  124. 124. Query at any time (q ’[:find ?email :where [_ :email ?email]] db) (q ’[:find ?email :where [_ :email ?email]] (as-of db #inst "2013-09-02")) onsdag 16 oktober 13
  125. 125. Query over time (q ’[:find ?e (count ?tx) :where [?e :email _ ?tx]] db) onsdag 16 oktober 13
  126. 126. Query over time (q ’[:find ?e (count ?tx) :where [?e :email _ ?tx]] db) (q ’[:find ?e (count ?tx) :where [?e :email _ ?tx]] (history db)) onsdag 16 oktober 13
  127. 127. Java API List<Object> results = Peer.q(someQuery, conn.db()); Future<Map> txResult = conn.transact(data_tx); onsdag 16 oktober 13
  128. 128. More info http://www.datomic.com/videos.html onsdag 16 oktober 13
  129. 129. Comparison onsdag 16 oktober 13
  130. 130. Comparison Data model Schema evolution Queries Recommendation onsdag 16 oktober 13
  131. 131. EventStore Events ✓ Understandable by normal people ✓ Designed for integration ✓ Forces aggregate design (scalability) ✓ Capture intent ๏ Require design effort to get right ๏ Complect transaction and query onsdag 16 oktober 13
  132. 132. Datomic Facts ✓ Support partial information ✓ Normalization support ✓ CRUD out of the box ๏ Static schema onsdag 16 oktober 13
  133. 133. EventStore schema evolution ✓ New events ✓ New event attributes ✓ Event attributes rename (change deserialization) ✓ Projected state change (rebuild projection) ๏ Cross event refactoring onsdag 16 oktober 13
  134. 134. Datomic schema evolution ✓ New or removed attributes ✓ New or removed entity types ✓ Move attributes between entities ๏ Fact attributes rename not possible ๏ Fact attribute type change not possible onsdag 16 oktober 13
  135. 135. EventStore queries ✓ Projections use JavaScript ✓ Persistent ๏ Require projection onsdag 16 oktober 13
  136. 136. Datomic queries ✓ Declarative ✓ Logic-based ✓ Allows calling out to your own code ๏ New language to learn onsdag 16 oktober 13
  137. 137. When to use Event Sourcing? Domains where events seem natural Different teams with different models onsdag 16 oktober 13
  138. 138. When to use Datomic? Data can be modelled using entities and attributes History is or may be interesting onsdag 16 oktober 13
  139. 139. More details http://www.jayway.com/author/jankronquist/ https://github.com/jankronquist http://www.slideshare.net/jankronquist onsdag 16 oktober 13
  140. 140. Questions? onsdag 16 oktober 13

×