Resilient Applications with Akka Persistence - Scaladays 2014

20,348 views

Published on

In this presentation you will learn how to leverage the features introduced in Akka Persistence: opt-in at-least-once delivery semantics between actors and the ability to recover application state after a crash. Both are implemented by storing immutable facts in a persisted append-only log. We will show you how to create persistent actors using command and event sourcing, replicate events with reliable communication, scale out and improve resilience with clustering.

  • Be the first to comment

Resilient Applications with Akka Persistence - Scaladays 2014

  1. 1. Resilient Applications with Akka Persistence Patrik Nordwall
 @patriknw Konrad Malawski
 @ktosopl Björn Antonsson
 @bantonsson
  2. 2. Reactive Applications Akka  Persistence  ScalaDays  2014
  3. 3. Resilient • Embrace Failure • Failure is a normal part of the application lifecycle • Self Heal • Failure is detected, isolated, and managed Akka  Persistence  ScalaDays  2014
  4. 4. The Naïve Way • Write State to Database • Transactions Everywhere • Problem Solved? • Not Scalable, Responsive, Event-Driven! Akka  Persistence  ScalaDays  2014
  5. 5. Command and Event Sourcing
  6. 6. Command and Event Sourcing • State is the sum of Events • Events are persisted to Store • Append only • Scales well Akka  Persistence  ScalaDays  2014
  7. 7. Command v.s. Event • Command • What someone wants me to do • Can be rejected • Event • Something that has already happened • An immutable fact Akka  Persistence  ScalaDays  2014
  8. 8. Commands can Generate Events • If I accept a Command and change State • Persist Event to Store • If I crash • Replay Events to recover State Akka  Persistence  ScalaDays  2014
  9. 9. Persist All Commands? • If I crash on a Command • I will likely crash during recovery • Like the Army • Don't question orders • Repeat until success Akka  Persistence  ScalaDays  2014
  10. 10. Only Persist Events • Only accepted Commands generate Events • No surprises during recovery • Like a dieting method • You are what you eat Akka  Persistence  ScalaDays  2014
  11. 11. Achievement Unlocked? • Resilient • State is recoverable • Scalable • Append only writes • Something Missing? • Queries Akka  Persistence  ScalaDays  2014
  12. 12. CQRS Command Query Responsibility Segregation
  13. 13. CQRS • Separate Models • Command Model • Optimized for command processing • Query Model • Optimized data presentation Akka  Persistence  ScalaDays  2014
  14. 14. Query Model from Events • Source the Events • Pick what fits • In Memory • SQL Database • Graph Database • Key Value Store Akka  Persistence  ScalaDays  2014
  15. 15. Akka  Persistence  ScalaDays  2014 Client Service Query   Model Command Store Query Store Command   Model
  16. 16. PersistentActor Akka  Persistence  ScalaDays  2014
  17. 17. PersistentActor Processor & Eventsourced Processor Replaces: in Akka 2.3.4+
  18. 18. super quick domain modelling! sealed trait Command! case class GiveMe(coins: Int) extends Command! case class TakeMy(coins: Int) extends Command Commands - what others “tell” us; not persisted case class Wallet(coins: Int) {! def updated(diff: Int) = State(coins + diff)! } State - reflection of a series of events sealed trait Event! case class BalanceChangedBy(coins: Int) extends Event! Events - reflect effects, past tense; persisted
  19. 19. var state = S0 ! def processorId = “a” ! PersistentActor Command ! ! Journal
  20. 20. PersistentActor var state = S0 ! def processorId = “a” ! ! ! Journal Generate Events
  21. 21. PersistentActor var state = S0 ! def processorId = “a” ! ! ! Journal Generate Events E1
  22. 22. PersistentActor ACK “persisted” ! ! Journal E1 var state = S0 ! def processorId = “a” !
  23. 23. PersistentActor “Apply” event ! ! Journal E1 var state = S1 ! def processorId = “a” ! E1
  24. 24. PersistentActor ! ! Journal E1 var state = S1 ! def processorId = “a” ! E1 Okey!
  25. 25. PersistentActor ! ! Journal E1 var state = S1 ! def processorId = “a” ! E1 Okey!
  26. 26. PersistentActor ! ! Journal E1 var state = S1 ! def processorId = “a” ! E1 Ok, he got my $.
  27. 27. PersistentActor class BitCoinWallet extends PersistentActor {! ! var state = Wallet(coins = 0)! ! def updateState(e: Event): State = {! case BalanceChangedBy(coins) => state.updatedWith(coins)! }! ! // API:! ! def receiveCommand = ??? // TODO! ! def receiveRecover = ??? // TODO! ! }!
  28. 28. persist(e) { e => }
  29. 29. PersistentActor def receiveCommand = {! ! case TakeMy(coins) =>! persist(BalanceChangedBy(coins)) { changed =>! state = updateState(changed) ! }! ! ! ! ! ! ! } async callback
  30. 30. PersistentActor: persist(){} def receiveCommand = {! ! ! ! ! ! ! case GiveMe(coins) if coins <= state.coins =>! persist(BalanceChangedBy(-coins)) { changed =>! state = updateState(changed) ! sender() ! TakeMy(coins)! }! } async callback Safe to mutate the Actor’s state
  31. 31. PersistentActor def receiveCommand = {! ! ! ! ! ! ! case GiveMe(coins) if coins <= state.coins =>! persist(BalanceChangedBy(-coins)) { changed =>! state = updateState(changed) ! sender() ! TakeMy(coins)! }! } Safe to access sender here
  32. 32. persist(){} - Ordering guarantees ! ! Journal E1 var state = S0 ! def processorId = “a” ! C1 C2 C3
  33. 33. ! ! Journal E1 var state = S0 ! def processorId = “a” ! C1 C2 C3 Commands get “stashed” until processing C1’s events are acted upon. persist(){} - Ordering guarantees
  34. 34. ! ! Journal var state = S0 ! def processorId = “a” ! C1 C2 C3 E1 E2 E2E1 events get applied in-order persist(){} - Ordering guarantees
  35. 35. C2 ! ! Journal var state = S0 ! def processorId = “a” ! C3 E1 E2 E2E1 and the cycle repeats persist(){} - Ordering guarantees
  36. 36. persistAsync(e) { e => }
  37. 37. persistAsync(e) { e => } + defer(e) { e => }
  38. 38. def receiveCommand = {! ! ! ! case Mark(id) =>! sender() ! InitMarking! persistAsync(Marker) { m =>! // update state...! }! ! ! ! ! } persistAsync PersistentActor: persistAsync(){} will NOT force stashing of commands
  39. 39. PersistentActor: persistAsync(){} def receiveCommand = {! ! ! ! case Mark(id) =>! sender() ! InitMarking! persistAsync(Marker) { m =>! // update state...! }! ! defer(Marked(id)) { marked =>! sender() ! marked! }! } execute once all persistAsync handlers done NOT persisted
  40. 40. persistAsync(){} - Ordering guarantees ! ! Journal var state = S0 ! def processorId = “a” ! C1 C2 C3
  41. 41. persistAsync(){} - Ordering guarantees ! ! Journal var state = S0 ! def processorId = “a” !C2 C3
  42. 42. persistAsync(){} - Ordering guarantees ! ! Journal var state = S0 ! def processorId = “a” ! C3
  43. 43. persistAsync(){} - Ordering guarantees ! ! Journal var state = S0 ! def processorId = “a” ! C3 E1 E2
  44. 44. persistAsync(){} - Ordering guarantees var state = S0 ! def processorId = “a” ! C3 E1 Akka  Persistence  ScalaDays ! ! Journal E1 E2
  45. 45. persistAsync(){} - Ordering guarantees E1 var state = S1 ! def processorId = “a” ! E2 E1 E2 ! ! Journal Akka  Persistence  ScalaDays E2 E3E1
  46. 46. persistAsync(){} - Ordering guarantees E1 var state = S2 ! def processorId = “a” ! E2 E1 E2 deferred handlers triggered M1 M2 ! ! Journal Akka  Persistence  ScalaDays E2 E3E1
  47. 47. Recovery Akka  Persistence  ScalaDays
  48. 48. Eventsourced, recovery /** MUST NOT SIDE-EFFECT! */! def receiveRecover = {! case replayedEvent: Event => ! state = updateState(replayedEvent)! } re-using updateState, as seen in receiveCommand Akka  Persistence  ScalaDays
  49. 49. Views Akka  Persistence  ScalaDays
  50. 50. Journal (DB) ! ! ! Views ! Processor ! def processorId = “a” ! polling Akka  Persistence  ScalaDays ! View ! def processorId = “a” ! ! !
  51. 51. Journal (DB) ! ! ! Views ! Processor ! def processorId = “a” ! polling ! View ! def processorId = “a” ! ! ! polling different ActorPath, same processorId Akka  Persistence  ScalaDays ! View ! def processorId = “a” ! ! !
  52. 52. View class DoublingCounterProcessor extends View {! var state = 0! 
 override val processorId = "counter"! ! def receive = {! case Persistent(payload, seqNr) =>! // “state += 2 * payload” ! ! }! } subject to change! Akka  Persistence  ScalaDays
  53. 53. Views, as Reactive Streams Akka  Persistence  ScalaDays
  54. 54. View, as ReactiveStream // Imports ...! ! import org.reactivestreams.api.Producer! ! import akka.stream._! import akka.stream.scaladsl.Flow! ! import akka.persistence._! import akka.persistence.stream._! 
 val materializer = FlowMaterializer(MaterializerSettings())! pull request by krasserm early preview Akka  Persistence  ScalaDays
  55. 55. View, as ReactiveStream // 1 producer and 2 consumers:! val p1: Producer[Persistent] = PersistentFlow.! fromProcessor(“processor-1").! toProducer(materializer)! ! Flow(p1).! foreach(p => println(s"consumer-1: ${p.payload}”)).! consume(materializer)! ! Flow(p1).! foreach(p => println(s"consumer-2: ${p.payload}”)).! consume(materializer) pull request by krasserm early preview Akka  Persistence  ScalaDays
  56. 56. View, as ReactiveStream // 2 producers (merged) and 1 consumer:! val p2: Producer[Persistent] = PersistentFlow.! fromProcessor(“processor-2").! toProducer(materializer)! 
 val p3: Producer[Persistent] = PersistentFlow.! fromProcessor(“processor-3").! toProducer(materializer)! ! Flow(p2).merge(p3). // triggers on “either”! foreach { p => println(s"consumer-3: ${p.payload}") }.! consume(materializer)! pull request by krasserm early preview Akka  Persistence  ScalaDays
  57. 57. Akka  Persistence  ScalaDays  2014 Usage in a Cluster • distributed journal (http://akka.io/community/) • Cassandra • DynamoDB • HBase • MongoDB • shared LevelDB journal for testing • single writer • cluster singleton • cluster sharding
  58. 58. Akka  Persistence  ScalaDays  2014 Cluster Singleton A B C D
  59. 59. Akka  Persistence  ScalaDays  2014 Cluster Singleton A B C D role: backend-1 role: backend-1 role: backend-2 role: backend-2
  60. 60. Akka  Persistence  ScalaDays  2014 Cluster Sharding A B C D
  61. 61. Akka  Persistence  ScalaDays  2014 Cluster Sharding sender id:17 region
 node-­‐1 coordinator region
 node-­‐2 region
 node-­‐3 GetShardHome:17 id:17 ShardHome:17  -­‐>  node2 17  -­‐>  node2
  62. 62. Akka  Persistence  ScalaDays  2014 Cluster Sharding sender region
 node-­‐1 coordinator region
 node-­‐2 region
 node-­‐3 id:17 id:17 GetShardHome:17 ShardHome:17  -­‐>  node2 id:17 17  -­‐>  node2 17  -­‐>  node2
  63. 63. Akka  Persistence  ScalaDays  2014 Cluster Sharding 17 sender region
 node-­‐1 coordinator region
 node-­‐2 region
 node-­‐3 id:17 id:17 17  -­‐>  node2 17  -­‐>  node2 17  -­‐>  node2
  64. 64. Akka  Persistence  ScalaDays  2014 Cluster Sharding 17 sender region
 node-­‐1 coordinator region
 node-­‐2 region
 node-­‐3 17  -­‐>  node2 17  -­‐>  node2 17  -­‐>  node2 id:17
  65. 65. Akka  Persistence  ScalaDays  2014 Cluster Sharding 17 sender region
 node-­‐1 coordinator region
 node-­‐2 region
 node-­‐3 17  -­‐>  node2 17  -­‐>  node2 17  -­‐>  node2 id:17
  66. 66. Cluster Sharding val idExtractor: ShardRegion.IdExtractor = { case cmd: Command => (cmd.postId, cmd) } ! val shardResolver: ShardRegion.ShardResolver = msg => msg match { case cmd: Command => (math.abs(cmd.postId.hashCode) % 100).toString } ClusterSharding(system).start( typeName = BlogPost.shardName, entryProps = Some(BlogPost.props()), idExtractor = BlogPost.idExtractor, shardResolver = BlogPost.shardResolver) val blogPostRegion: ActorRef = 
 ClusterSharding(context.system).shardRegion(BlogPost.shardName) ! val postId = UUID.randomUUID().toString blogPostRegion ! BlogPost.AddPost(postId, author, title)
  67. 67. Akka  Persistence  ScalaDays  2014 Lost messages sender destination $
  68. 68. Akka  Persistence  ScalaDays  2014 At-least-once delivery - duplicates sender destination $ ok $ $ $ ok Re-­‐send
  69. 69. Akka  Persistence  ScalaDays  2014 M2 At-least-once delivery - unordered sender destination M1 ok  1 ok  2 M2 ok  3 M3 M1M3 M2 Re-­‐send
  70. 70. Akka  Persistence  ScalaDays  2014 M2 At-least-once delivery - crash sender destination M1 ok  1 ok  2 M2 ok  3 M3 1. Sent  M1   2. Sent  M2   3. Sent  M3   M3 5.  M2  Confirmed   6.  M3  Confirmed 4.  M1  Confirmed sender M1M2 M3
  71. 71. PersistentActor with AtLeastOnceDelivery case class Msg(deliveryId: Long, s: String) case class Confirm(deliveryId: Long) sealed trait Evt case class MsgSent(s: String) extends Evt case class MsgConfirmed(deliveryId: Long) extends Evt class Sender(destination: ActorPath) extends PersistentActor with AtLeastOnceDelivery { ! def receiveCommand: Receive = { case s: String => persist(MsgSent(s))(updateState) case Confirm(deliveryId) => persist(MsgConfirmed(deliveryId))(updateState) } ! def receiveRecover: Receive = { case evt: Evt => updateState(evt) } ! def updateState(evt: Evt): Unit = evt match { case MsgSent(s) => deliver(destination, deliveryId => Msg(deliveryId, s)) ! case MsgConfirmed(deliveryId) => confirmDelivery(deliveryId) } }
  72. 72. Akka  Persistence  ScalaDays  2014 Next step • Documentation • http://doc.akka.io/docs/akka/2.3.3/scala/persistence.html • http://doc.akka.io/docs/akka/2.3.3/java/persistence.html • http://doc.akka.io/docs/akka/2.3.3/contrib/cluster-sharding.html • Typesafe Activator • https://typesafe.com/activator/template/akka-sample-persistence-scala • https://typesafe.com/activator/template/akka-sample-persistence-java • http://typesafe.com/activator/template/akka-cluster-sharding-scala • Mailing list • http://groups.google.com/group/akka-user • Migration guide from Eventsourced • http://doc.akka.io/docs/akka/2.3.3/project/migration-guide-eventsourced-2.3.x.html
  73. 73. ©Typesafe 2014 – All Rights Reserved

×