JavaZone 2013 - Datomic vs EventStore

4,480 views

Published on

If you are used to traditional databases, then to only append and never update your data may sound like a crazy idea. However, not only does this enable historical queries, but also enhances fault tolerance and scalability. In this presentation we briefly describe two immutable data stores (Rich Hickey's Datomic and Greg Young's EventStore) and compare their different data models using an example problem domain. Along the way we learn about CQRS, Aggregates, Projections and why you want your data to be immutable.

EventStore is a data store for applications using event sourcing and time-series data. EventStore runs on .NET and Mono.

Datomic is a database of time-based facts, with declarative queries and ACID transactions. Datomic is written in Clojure and runs on the JVM.

JavaZone 2013 - Datomic vs EventStore

  1. 1. vs Immutable data stores onsdag 25 september 13
  2. 2. What? onsdag 25 september 13
  3. 3. Immutable data stores Only add, never remove or change Can retrieve old values onsdag 25 september 13
  4. 4. Source code Version controlled Keep all versions onsdag 25 september 13
  5. 5. Example domain onsdag 25 september 13
  6. 6. Rock - Paper - Scissors onsdag 25 september 13
  7. 7. http://rock-paper-scissors.com/ The future Facebook of Rock Paper Scissors Millions of users Many games per user onsdag 25 september 13
  8. 8. Playing the game Player A Player B Server onsdag 25 september 13
  9. 9. Playing the game Player A Player B Server Game 123 onsdag 25 september 13
  10. 10. Playing the game Player A Player B Server paper Game 123 onsdag 25 september 13
  11. 11. Playing the game Player A Player B Server paper Player B: paper Game 123 onsdag 25 september 13
  12. 12. Playing the game Player A Player B Server rock Player B: paper Player A: rock Game 123 onsdag 25 september 13
  13. 13. Playing the game Player A Player B Server Player B: paper Player A: rock Game 123 Game 123 winner: Player B loser: Player A onsdag 25 september 13
  14. 14. Playing the game Player A Player B Server Player B: paper Player A: rock Game 123 Game 123 winner: Player B loser: Player A CREATED WAITING GAME WON GAME TIED any move other move (victory) other move (tie) onsdag 25 september 13
  15. 15. Entities onsdag 25 september 13
  16. 16. Entities Player + id + name + email onsdag 25 september 13
  17. 17. Entities Game + id + state + players + moves + winner + loser Player + id + name + email onsdag 25 september 13
  18. 18. Entities Game + id + state + players + moves + winner + loser Move + move + player Player + id + name + email onsdag 25 september 13
  19. 19. Actions Create game Make move Get game result POST /games POST /games/123 move=rock GET /games/123 onsdag 25 september 13
  20. 20. Actions Create game Make move Get game result POST /games POST /games/123 move=rock GET /games/123 1. Form (make move) 2. Waiting 3. Game result 4. Error onsdag 25 september 13
  21. 21. Event Sourcing with Event Store onsdag 25 september 13
  22. 22. Event Sourcing Events that have happened Ordered in time Source of truth onsdag 25 september 13
  23. 23. http://geteventstore.com Runs on .NET and Mono Free (store) or Commercial (HA) API Custom TCP (c#) ATOM over HTTP (xml/json) Features Event streams, projections, generate new events (CEP) €1500/year onsdag 25 september 13
  24. 24. Event example Server EventStore onsdag 25 september 13
  25. 25. Event example Server CreateGameCommand{:game 123 :creator "player-1"} EventStore onsdag 25 september 13
  26. 26. Event example Server CreateGameCommand{:game 123 :creator "player-1"} GameCreatedEvent{:game 123 :creator "player-1"} EventStore onsdag 25 september 13
  27. 27. Event example Server MakeMoveCommand{:game 123 :player "player-1" :move "rock"} GameCreatedEvent{:game 123 :creator "player-1"} MoveMadeEvent{:game 123 :player "player-1" :move "rock"} EventStore onsdag 25 september 13
  28. 28. Event example Server MakeMoveCommand{:game 123 :player "player-2" :move "paper"} GameCreatedEvent{:game 123 :creator "player-1"} MoveMadeEvent{:game 123 :player "player-1" :move "rock"} MoveMadeEvent{:game 123 :player "player-2" :move "paper"} GameWonEvent{:game 123 :winner "player-2" :loser "player-1"} EventStore onsdag 25 september 13
  29. 29. GameCreatedEvent{:game 123, :creator "player-1", :time 1} MoveMadeEvent{:game 123, :player "player-1", :move "rock", :time 2} Current state? onsdag 25 september 13
  30. 30. CREATED WAITING GAME WON GAME TIED any move if victory other move if tie GameCreatedEvent{:game 123, :creator "player-1", :time 1} MoveMadeEvent{:game 123, :player "player-1", :move "rock", :time 2} Current state? onsdag 25 september 13
  31. 31. CREATED WAITING GAME WON GAME TIED any move if victory other move if tie GameCreatedEvent{:game 123, :creator "player-1", :time 1} MoveMadeEvent{:game 123, :player "player-1", :move "rock", :time 2} GameCreatedEvent Current state? CREATED onsdag 25 september 13
  32. 32. CREATED WAITING GAME WON GAME TIED any move if victory other move if tie GameCreatedEvent{:game 123, :creator "player-1", :time 1} MoveMadeEvent{:game 123, :player "player-1", :move "rock", :time 2} Current state? WAITING MoveMadeEvent onsdag 25 september 13
  33. 33. Aggregates Scope of consistency when handling a command Game + id + state + players + moves + winner + loser Move + id + move + player Player + id + name + email onsdag 25 september 13
  34. 34. Aggregates as Event Streams event1 GameCreatedEvent{:game 1...} event2 MoveMadeEvent{:game 1....} stream = game-1 version = 2 onsdag 25 september 13
  35. 35. Aggregates as Event Streams event1 GameCreatedEvent{:game 1...} event2 MoveMadeEvent{:game 1....} stream = game-1 version = 2 event1 GameCreatedEvent{:game 2...} event2 MoveMadeEvent{:game 2....} event3 MoveMadeEvent{:game 2....} event4 GameWonEvent{:game 2....} stream = game-2 version = 4 onsdag 25 september 13
  36. 36. Aggregates as Event Streams event1 GameCreatedEvent{:game 1...} event2 MoveMadeEvent{:game 1....} stream = game-1 version = 2 event1 GameCreatedEvent{:game 2...} event2 MoveMadeEvent{:game 2....} event3 MoveMadeEvent{:game 2....} event4 GameWonEvent{:game 2....} stream = game-2 version = 4 curl "http://127.0.0.1:2113/streams/game-1" -d @eventdata.json -H "Content-Type:application/json" -H "ES-ExpectedVersion: 2" onsdag 25 september 13
  37. 37. Projections GameCreatedEvent{:game 1...} MoveMadeEvent{:game 1....} GameCreatedEvent{:game 2...} MoveMadeEvent{:game 2....} MoveMadeEvent{:game 2....} GameWonEvent{:game 2....} onsdag 25 september 13
  38. 38. Projections GameCreatedEvent{:game 1...} MoveMadeEvent{:game 1....} GameCreatedEvent{:game 2...} MoveMadeEvent{:game 2....} MoveMadeEvent{:game 2....} GameWonEvent{:game 2....} for (Event events : allEvents) { if (event instanceof GameCreatedEvent) { created++; } } return created; onsdag 25 september 13
  39. 39. Projections GameCreatedEvent{:game 1...} MoveMadeEvent{:game 1....} GameCreatedEvent{:game 2...} MoveMadeEvent{:game 2....} MoveMadeEvent{:game 2....} GameWonEvent{:game 2....} onsdag 25 september 13
  40. 40. Projections GameCreatedEvent{:game 1...} MoveMadeEvent{:game 1....} GameCreatedEvent{:game 2...} MoveMadeEvent{:game 2....} MoveMadeEvent{:game 2....} GameWonEvent{:game 2....} created = 2 onsdag 25 september 13
  41. 41. Projections GameCreatedEvent{:game 1...} MoveMadeEvent{:game 1....} GameCreatedEvent{:game 2...} MoveMadeEvent{:game 2....} MoveMadeEvent{:game 2....} GameWonEvent{:game 2....} created = 2 GameCreatedEvent{:game 3...} onsdag 25 september 13
  42. 42. Projections GameCreatedEvent{:game 1...} MoveMadeEvent{:game 1....} GameCreatedEvent{:game 2...} MoveMadeEvent{:game 2....} MoveMadeEvent{:game 2....} GameWonEvent{:game 2....} (state, event) -> { if (event instanceof GameCreatedEvent) { return state++; } rerturn state; } created = 2 GameCreatedEvent{:game 3...} onsdag 25 september 13
  43. 43. Projections in Javascript fromAll() .when({ $init: function () { return { created: 0, completed:0 }; }, }); onsdag 25 september 13
  44. 44. Projections in Javascript fromAll() .when({ $init: function () { return { created: 0, completed:0 }; }, }); GameCreatedEvent: function(s, e) { s.created++; return s; }, onsdag 25 september 13
  45. 45. Projections in Javascript fromAll() .when({ $init: function () { return { created: 0, completed:0 }; }, }); GameCreatedEvent: function(s, e) { s.created++; return s; }, GameWonEvent: function(s, e) { s.completed++; return s; }, onsdag 25 september 13
  46. 46. Projections in Javascript fromAll() .when({ $init: function () { return { created: 0, completed:0 }; }, }); GameCreatedEvent: function(s, e) { s.created++; return s; }, GameWonEvent: function(s, e) { s.completed++; return s; }, GameTiedEvent: function(s, e) { s.completed++; return s; } onsdag 25 september 13
  47. 47. Projections in Javascript fromAll() .when({ $init: function () { return { created: 0, completed:0 }; }, }); GameCreatedEvent: function(s, e) { s.created++; return s; }, GameWonEvent: function(s, e) { s.completed++; return s; }, GameTiedEvent: function(s, e) { s.completed++; return s; } /projections/games-counter { created: 15, completed: 10 } onsdag 25 september 13
  48. 48. External Projections events Business logic onsdag 25 september 13
  49. 49. External Projections events events Business logic Projections onsdag 25 september 13
  50. 50. External Projections Consistent Available events events Business logic Projections onsdag 25 september 13
  51. 51. External Projections Commands Queries Consistent Available events events onsdag 25 september 13
  52. 52. System with EventStore onsdag 25 september 13
  53. 53. System with EventStore Command Handler onsdag 25 september 13
  54. 54. System with EventStore 1.command Command Handler onsdag 25 september 13
  55. 55. System with EventStore 1.command Event Store Command Handler 2.get events onsdag 25 september 13
  56. 56. System with EventStore 1.command Event Store Command Handler 2.get events 3.add events onsdag 25 september 13
  57. 57. System with EventStore 1.command Event Store Command Handler 2.get events 3.add events 4.update projections onsdag 25 september 13
  58. 58. System with EventStore 1.command Event Store Command Handler 2.get events 3.add events 4.update projections 5.get events or projections Some service onsdag 25 september 13
  59. 59. System with EventStore 1.command Event Store Command Handler 2.get events 6.update state 3.add events 4.update projections 5.get events or projections Some service onsdag 25 september 13
  60. 60. System with EventStore 1.command Event Store Command Handler 2.get events 6.update state 3.add events 4.update projections 7.query 5.get events or projections Some service onsdag 25 september 13
  61. 61. Datomic onsdag 25 september 13
  62. 62. http://www.datomic.com/ Runs on JVM Free (3 peers, local storage) or Commercial (HA, distrib) Features ACID transactions, declarative query engine €2300 onsdag 25 september 13
  63. 63. Fact Something known to have happened or existed onsdag 25 september 13
  64. 64. Fact Something known to have happened or existed 2004-10-01 the email of Jan was ”jan.kronquist@jayway.se” onsdag 25 september 13
  65. 65. Fact Something known to have happened or existed 2004-10-01 the email of Jan was ”jan.kronquist@jayway.se” 2007-01-01 the email of Jan was ”jan.kronquist@jayway.com” onsdag 25 september 13
  66. 66. Atomic fact (datom) Entity Attribute Value Time onsdag 25 september 13
  67. 67. Atomic fact (datom) Entity Attribute Value Time The person Jan Kronquist email ”jan.kronquist@jayway.se” 2004-10-01 onsdag 25 september 13
  68. 68. Data perception Facts Add facts Add more facts Even more facts time onsdag 25 september 13
  69. 69. Data perception Facts Add facts Add more facts Even more facts time onsdag 25 september 13
  70. 70. Data perception Facts Add facts Add more facts Even more facts time query onsdag 25 september 13
  71. 71. Data perception Facts Add facts Add more facts Even more facts time query query onsdag 25 september 13
  72. 72. Data perception Facts Add facts Add more facts Even more facts time query onsdag 25 september 13
  73. 73. Data perception Facts Add facts Add more facts Even more facts time query onsdag 25 september 13
  74. 74. Data perception Facts Add facts Add more facts Even more facts time query onsdag 25 september 13
  75. 75. Database deconstructed onsdag 25 september 13
  76. 76. Database deconstructed Transactor Consistency by serializing transactions onsdag 25 september 13
  77. 77. Database deconstructed Transactor Consistency by serializing transactions Peer Your code + Query engine onsdag 25 september 13
  78. 78. Database deconstructed Transactor Consistency by serializing transactions Peer Your code + Query engine Storage service Key/value store Scalable reads and writes eg DynamoDB, InfiniSpan, Riak + ZooKeeper, SQL(!) onsdag 25 september 13
  79. 79. System with Datomic Transactor Distributed Storage Service Peer APeer A onsdag 25 september 13
  80. 80. System with Datomic Transactor Distributed Storage Service Peer APeer A 1.command onsdag 25 september 13
  81. 81. System with Datomic Transactor Distributed Storage Service Peer APeer A 2.read and cache 1.command onsdag 25 september 13
  82. 82. System with Datomic Transactor Distributed Storage Service Peer A 3.transact Peer A 2.read and cache 1.command onsdag 25 september 13
  83. 83. System with Datomic Transactor Distributed Storage Service Peer A 3.transact Peer A 4.write 2.read and cache 1.command onsdag 25 september 13
  84. 84. System with Datomic Transactor Distributed Storage Service Peer A 3.transact Peer A 4.write 2.read and cache Peer B 5.notify 1.command onsdag 25 september 13
  85. 85. Datomic example Peer Transactor onsdag 25 september 13
  86. 86. Datomic example Peer CreateGameCommand{:game 123 :creator "player-1"} Transactor onsdag 25 september 13
  87. 87. Datomic example Peer CreateGameCommand{:game 123 :creator "player-1"} [123 :creator "player-1" T1] [123 :state :created T1] Transactor onsdag 25 september 13
  88. 88. Datomic example Peer CreateGameCommand{:game 123 :creator "player-1"} [123 :creator "player-1" T1] [123 :state :created T1] Transactor ["player-1" :email "player@gmail.com" T0] onsdag 25 september 13
  89. 89. Datomic example Peer CreateGameCommand{:game 123 :creator "player-1"} [123 :creator "player-1" T1] [123 :state :created T1] Transactor onsdag 25 september 13
  90. 90. Datomic example Peer CreateGameCommand{:game 123 :creator "player-1"} [123 :creator "player-1" T1] [123 :state :created T1] Transactor GameCreatedEvent{:game 123 :creator "player-1"} onsdag 25 september 13
  91. 91. Datomic example Peer CreateGameCommand{:game 123 :creator "player-1"} [123 :creator "player-1" T1] [123 :state :created T1] Transactor onsdag 25 september 13
  92. 92. Datomic example Peer MakeMoveCommand{:game 123 :player "player-1" :move :rock} [123 :creator "player-1" T1] [123 :state :created T1] [123_1 :move :rock T2] [123_1 :player "player-1" T2] [123 :moves 123_1 T2] [123 :state :waiting T2] Transactor onsdag 25 september 13
  93. 93. Datomic example Peer MakeMoveCommand{:game 123 :player "player-2" :move :paper} [123 :creator "player-1" T1] [123 :state :created T1] [123_1 :move :rock T2] [123_1 :player "player-1" T2] [123 :moves 123_1 T2] [123 :state :waiting T2] [123_2 :move :paper T3] [123_2 :player "player-2" T3] [123 :moves 123_2 T3] [123 :state :won T3] [123 :winner "player-2" T3] [123 :loser "player-1" T3] Transactor onsdag 25 september 13
  94. 94. Aggregates? Only by convention cas (compare and set) isComponent on relation Game + id + state + players + moves + winner + loser Move + id + move + player onsdag 25 september 13
  95. 95. What to find Where clauses [entity attribute value] Implicit joins Call arbitrary function Query language onsdag 25 september 13
  96. 96. Query example - basic [:find ?email :where [_ :email ?email]] onsdag 25 september 13
  97. 97. Query example - basic [:find ?email :where [_ :email ?email]] [["jan.kronquist@jayway.com] ["player-1@gmail.com"] ["player-2@gmail.com"]] onsdag 25 september 13
  98. 98. Query example - aggregation [:find ?state (?count game) :where [?game :state ?state]] onsdag 25 september 13
  99. 99. Query example - aggregation [:find ?state (?count game) :where [?game :state ?state]] [[:created 4] [:waiting 1] [:won 7] [:tied 3]] onsdag 25 september 13
  100. 100. Query example - aggregation [:find ?state (?count game) :where [?game :state ?state]] [[:created 4] [:waiting 1] [:won 7] [:tied 3]] inProgess = created + waiting onsdag 25 september 13
  101. 101. Query example - aggregation [:find ?state (?count game) :where [?game :state ?state]] [[:created 4] [:waiting 1] [:won 7] [:tied 3]] inProgess = created + waiting fromAll() .when({ $init: function () { return { created: 0, completed:0 }; }, GameCreatedEvent: function(s, e) { s.created++; return s; }, GameWonEvent: function(s, e) { s.completed++; return s; }, GameTiedEvent: function(s, e) { s.completed++; return s; } }); /projections/games-counter { created: 15, completed: 10 } onsdag 25 september 13
  102. 102. Query example - join [:find ?email (count ?game) :where [?player :email ?email] [?game :winner ?player]] onsdag 25 september 13
  103. 103. Query at any time (q ’[:find ?email :where [_ :email ?email]] db) onsdag 25 september 13
  104. 104. Query at any time (q ’[:find ?email :where [_ :email ?email]] db) (q ’[:find ?email :where [_ :email ?email]] (as-of db #inst "2013-09-02")) onsdag 25 september 13
  105. 105. Query over time (q ’[:find ?e (count ?tx) :where [?e :email _ ?tx]] db) onsdag 25 september 13
  106. 106. Query over time (q ’[:find ?e (count ?tx) :where [?e :email _ ?tx]] db) (q ’[:find ?e (count ?tx) :where [?e :email _ ?tx]] (history db)) onsdag 25 september 13
  107. 107. Java API List<Object> results = Peer.q(someQuery, conn.db()); Future<Map> txResult = conn.transact(data_tx); onsdag 25 september 13
  108. 108. Comparison onsdag 25 september 13
  109. 109. Comparison Data model Schema evolution Queries Recommendation onsdag 25 september 13
  110. 110. EventStore Events ✓ Understandable by normal people ✓ Designed for integration ✓ Forces aggregate design (scalability) ✓ Capture intent ๏ Require design effort to get right ๏ Complect transaction and query onsdag 25 september 13
  111. 111. Datomic Facts ✓ Support partial information ✓ Normalization support ✓ CRUD out of the box ๏ Static schema onsdag 25 september 13
  112. 112. ✓ New events ✓ New event attributes ✓ Event attributes rename (change deserialization) ✓ Projected state change (rebuild projection) ๏ Cross event refactoring EventStore schema evolution onsdag 25 september 13
  113. 113. ✓ New or removed attributes ✓ New or removed entity types ✓ Move attributes between entities ๏ Fact attributes rename not possible ๏ Fact attribute type change not possible Datomic schema evolution onsdag 25 september 13
  114. 114. ✓ Projections use JavaScript ✓ Persistent ๏ Require projection EventStore queries onsdag 25 september 13
  115. 115. ✓ Declarative ✓ Logic-based ✓ Allows calling out to your own code ๏ New language to learn Datomic queries onsdag 25 september 13
  116. 116. When to use Event Sourcing? Domains where events seem natural Different teams with different models onsdag 25 september 13
  117. 117. When to use Datomic? Data can be modelled using entities and attributes History is or may be interesting onsdag 25 september 13
  118. 118. More details http://www.jayway.com/author/jankronquist/ https://github.com/jankronquist onsdag 25 september 13
  119. 119. Questions? onsdag 25 september 13

×