Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Islands in the Stream: Integrating Akka Streams and Akka Actors

85 views

Published on

After realizing the power of the Akka Streams API for building scalable, reliable, and resilient systems for streaming data, people are often confused as to how the Akka Streams API relates to Akka Actors. I will demonstrate how actors and streams solve complimentary problems. I will start with the basic patterns for interfacing both streams with actors, and actors with streams. Then I will show how actors compliment streams with regard to error handling, location transparency, and the distribution of workloads. I will demonstrate a large number of streaming workloads running in an Akka Cluster, by leveraging Cluster Sharding and Distributed Data. I will conclude by showing how an actor can use Akka Persistence to implement an Event Sourcing model for an Akka Stream. Understanding how to integrate Akka Actors and the Akka Streams API is essential for building scalable, reliable, and resilient distributed-applications to handle streaming workloads.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Islands in the Stream: Integrating Akka Streams and Akka Actors

  1. 1. ISLANDS IN THE STREAM INTEGRATING AKKA STREAMS AND ACTORS
  2. 2. HOW MUCH POWER ARE THESE ASSETS PRODUCING RIGHT NOW?
  3. 3. PART I - A MOTIVATING EXAMPLE > Actors > Streams
  4. 4. PART II - INTEGRATING ACTORS AND STREAMS > Messaging > Lifecycle Management > Distributing Workloads > Location Transparent Query > Persistence
  5. 5. PART I : A MOTIVATING EXAMPLE
  6. 6. PROOF OF CONCEPT
  7. 7. class Database extends Actor { val client = new DatabaseClient() def receive = { case Insert(sample) => client.insertAsync(sample) } }
  8. 8. class Database extends Actor { val client = new DatabaseClient() def receive = { case Insert(sample) => client.insertAsync(sample) } }
  9. 9. class Database extends Actor { val client = new DatabaseClient() def receive = { case Insert(sample) => client.insertAsync(sample) } }
  10. 10. class Database extends Actor { val client = new DatabaseClient() def receive = { case Insert(sample) => client.insertAsync(sample) } }
  11. 11. class Database extends Actor { val client = new DatabaseClient() def receive = { case Insert(sample) => client.insertAsync(sample) } }
  12. 12. val database = system.actorOf(Props[Database], "database") val websocket = Flow[Message] .collect { case TextMessage.Strict(text) => val message = Insert.parse(text) database ! message } .mapConcat(_ => Nil)
  13. 13. val database = system.actorOf(Props[Database], "database") val websocket = Flow[Message] .collect { case TextMessage.Strict(text) => val message = Insert.parse(text) database ! message } .mapConcat(_ => Nil)
  14. 14. val database = system.actorOf(Props[Database], "database") val websocket = Flow[Message] .collect { case TextMessage.Strict(text) => val message = Insert.parse(text) database ! message } .mapConcat(_ => Nil)
  15. 15. GROUPING
  16. 16. class Database extends Actor { val client = new DatabaseClient() var samples: Seq[Sample] = Nil var count: Int = 0 def receive = { case Insert(sample) => samples = sample +: samples count += 1 if (count == 1000) { insert() } } def insert() = { client.bulkInsertAsync(samples) samples = Nil count = 0 } }
  17. 17. class Database extends Actor { val client = new DatabaseClient() var samples: Seq[Sample] = Nil var count: Int = 0 def receive = { case Insert(sample) => samples = sample +: samples count += 1 if (count == 1000) { insert() } } def insert() = { client.bulkInsertAsync(samples) samples = Nil count = 0 } }
  18. 18. class Database extends Actor { val client = new DatabaseClient() var samples: Seq[Sample] = Nil var count: Int = 0 def receive = { case Insert(sample) => samples = sample +: samples count += 1 if (count == 1000) { insert() } } def insert() = { client.bulkInsertAsync(samples) samples = Nil count = 0 } }
  19. 19. class Database extends Actor { val client = new DatabaseClient() var samples: Seq[Sample] = Nil var count: Int = 0 def receive = { case Insert(sample) => samples = sample +: samples count += 1 if (count == 1000) { insert() } } def insert() = { client.bulkInsertAsync(samples) samples = Nil count = 0 } }
  20. 20. class Database extends Actor { val client = new DatabaseClient() var samples: Seq[Sample] = Nil var count: Int = 0 def receive = { case Insert(sample) => samples = sample +: samples count += 1 if (count == 1000) { insert() } } def insert() = { client.bulkInsertAsync(samples) samples = Nil count = 0 } }
  21. 21. PERIODIC EVENTS
  22. 22. class Database extends Actor with Timers { val client = new DatabaseClient() var samples: Seq[Sample] = Nil var count: Int = 0 timers.startSingleTimer(TickKey, Tick, 1 second) def receive = { case Insert(sample) => samples = sample +: samples count += 1 if (count == 1000) { insert() } case Tick => insert() timers.startSingleTimer(TickKey, Tick, 1 second) } def insert() = { if (count > 0) { client.bulkInsertAsync(samples) samples = Nil count = 0 } } }
  23. 23. class Database extends Actor with Timers { val client = new DatabaseClient() var samples: Seq[Sample] = Nil var count: Int = 0 timers.startSingleTimer(TickKey, Tick, 1 second) def receive = { case Insert(sample) => samples = sample +: samples count += 1 if (count == 1000) { insert() } case Tick => insert() timers.startSingleTimer(TickKey, Tick, 1 second) } def insert() = { if (count > 0) { client.bulkInsertAsync(samples) samples = Nil count = 0 } } }
  24. 24. Premature optimization is the root of all evil — Donald Knuth
  25. 25. class Database extends Actor with Timers { val client = new DatabaseClient() var samples: Seq[Sample] = Nil var count: Int = 0 var flush = true timers.startSingleTimer(TickKey, Tick, 1 second) def receive = { case Insert(sample) => samples = sample +: samples count += 1 if (count == 1000) { insert() flush = false } case Tick => if (flush) insert() else flush = true timers.startSingleTimer(TickKey, Tick, 1 second) } def insert() = { if (count > 0) { client.bulkInsertAsync(samples) samples = Nil count = 0 } } }
  26. 26. var flush = true def receive = { case Insert(sample) => samples = sample +: samples count += 1 if (count == 1000) { insert() flush = false } case Tick => if (flush) insert() else flush = true timers.startSingleTimer(TickKey, Tick, 1 second) }
  27. 27. var flush = true def receive = { case Insert(sample) => samples = sample +: samples count += 1 if (count == 1000) { insert() flush = false } case Tick => if (flush) insert() else flush = true timers.startSingleTimer(TickKey, Tick, 1 second) }
  28. 28. var flush = true def receive = { case Insert(sample) => samples = sample +: samples count += 1 if (count == 1000) { insert() flush = false } case Tick => if (flush) insert() else flush = true timers.startSingleTimer(TickKey, Tick, 1 second) }
  29. 29. RATE LIMITING REQUESTS
  30. 30. var outstanding = 0 def insert() = { if (count > 0 && outstanding < 10) { outstanding += 1 val (insert, remaining) = samples.splitAt(1000) samples = remaining count = remaining.size client.bulkInsertAsync(insert) andThen { case _ => self ! Decrement } } }
  31. 31. var outstanding = 0 def insert() = { if (count > 0 && outstanding < 10) { outstanding += 1 val (insert, remaining) = samples.splitAt(1000) samples = remaining count = remaining.size client.bulkInsertAsync(insert) andThen { case _ => self ! Decrement } } }
  32. 32. var outstanding = 0 def insert() = { if (count > 0 && outstanding < 10) { outstanding += 1 val (insert, remaining) = samples.splitAt(1000) samples = remaining count = remaining.size client.bulkInsertAsync(insert) andThen { case _ => self ! Decrement } } }
  33. 33. var outstanding = 0 def insert() = { if (count > 0 && outstanding < 10) { outstanding += 1 val (insert, remaining) = samples.splitAt(1000) samples = remaining count = remaining.size client.bulkInsertAsync(insert) andThen { case _ => self ! Decrement } } }
  34. 34. def receive = { case Insert(sample) => samples = sample +: samples count += 1 if (count == 1000) { insert() flush = false } case Tick => if (flush) insert() else flush = true timers.startSingleTimer(TickKey, Tick, 1 second) case Decrement => outstanding -= 1 if (count >= 1000) { insert() flush = false } }
  35. 35. def receive = { case Insert(sample) => samples = sample +: samples count += 1 if (count == 1000) { insert() flush = false } case Tick => if (flush) insert() else flush = true timers.startSingleTimer(TickKey, Tick, 1 second) case Decrement => outstanding -= 1 if (count >= 1000) { insert() flush = false } }
  36. 36. PROOF OF CONCEPT
  37. 37. class Database extends Actor { val client = new DatabaseClient() def receive = { case Insert(sample) => client.insertAsync(sample) } }
  38. 38. class Database extends Actor with Timers { val client = new DatabaseClient() var samples: Seq[Sample] = Nil var count: Int = 0 var flush = true var outstanding = 0 timers.startSingleTimer(TickKey, Tick, 1 second) def receive = { case Insert(sample) => samples = sample +: samples count += 1 if (count == 1000) { insert() flush = false } case Tick => if (flush) insert() else flush = true timers.startSingleTimer(TickKey, Tick, 1 second) case Decrement => outstanding -= 1 if (count >= 1000) { insert() flush = false } } def insert() = { if (count > 0 && outstanding < 10) { outstanding += 1 val (insert, remaining) = samples.splitAt(1000) samples = remaining count = remaining.size client.bulkInsertAsync(insert) andThen { case _ => self ! Decrement } } } }
  39. 39. def insert() = { if (count > 0 && outstanding < 10) { outstanding += 1 val (insert, remaining) = samples.splitAt(1000) samples = remaining count = remaining.size client.bulkInsertAsync(insert) andThen { case _ => self ! Decrement } } }
  40. 40. ERROR HANDLING
  41. 41. java.lang.OutOfMemoryError! Credit: Jonas Bonér
  42. 42. PROOF OF CONCEPT
  43. 43. AKKA STREAMS API
  44. 44. SOURCE
  45. 45. SINK
  46. 46. ELEMENT
  47. 47. FLOW
  48. 48. GRAPH
  49. 49. BACKPRESSURE
  50. 50. PATTERNSFOR STREAMING TELEMETRY
  51. 51. GROUPING
  52. 52. grouped
  53. 53. groupedWithin
  54. 54. RATE LIMITING REQUESTS
  55. 55. mapAsync(2)
  56. 56. mapAsync(4)
  57. 57. mapAsync(4)
  58. 58. THROTTLING
  59. 59. throttle
  60. 60. PROOF OF CONCEPT
  61. 61. val websocket = Flow[Message] .collect { case TextMessage.Strict(text) => val message = Insert.parse(text) database ! message } .mapConcat(_ => Nil)
  62. 62. AKKA STREAMS API val websocket = Flow[Message] .collect { case TextMessage.Strict(text) => InsertMessage.parse(text) } .groupedWithin(1000, 1 second) .mapAsync(10)(database.bulkInsertAsync) .map(messages => Message.ack(messages.last))
  63. 63. class Database extends Actor with Timers { val client = new DatabaseClient() var samples: Seq[Sample] = Nil var count: Int = 0 var flush = true var outstanding = 0 timers.startSingleTimer(TickKey, Tick, 1 second) def receive = { case Insert(sample) => samples = sample +: samples count += 1 if (count == 1000) { insert() flush = false } case Tick => if (flush) insert() else flush = true timers.startSingleTimer(TickKey, Tick, 1 second) case Decrement => outstanding -= 1 if (count >= 1000) { insert() flush = false } } def insert() = { if (count > 0 && outstanding < 10) { outstanding += 1 val (insert, remaining) = samples.splitAt(1000) samples = remaining count = remaining.size client.bulkInsertAsync(insert) andThen { case _ => self ! Decrement } } } }
  64. 64. val websocket = Flow[Message] .collect { case TextMessage.Strict(text) => InsertMessage.parse(text) } .groupedWithin(1000, 1 second) .mapAsync(10)(database.bulkInsertAsync) .map(messages => Message.ack(messages.last))
  65. 65. PROOF OF CONCEPT
  66. 66. PART II : ACTORSAND STREAMS
  67. 67. BRIDGING STREAMS AND ACTORS
  68. 68. mapAsync
  69. 69. class Sum extends Actor { var sum: Long = 0 def receive = { case Increment(value) => sum = sum + value case Query => CurrentSum(sum) } }
  70. 70. class Sum extends Actor { var sum: Long = 0 def receive = { case Increment(value) => sum = sum + value case Query => CurrentSum(sum) } }
  71. 71. class Sum extends Actor { var sum: Long = 0 def receive = { case Increment(value) => sum = sum + value case Query => CurrentSum(sum) } }
  72. 72. class Sum extends Actor { var sum: Long = 0 def receive = { case Increment(value) => sum = sum + value case Query => CurrentSum(sum) } }
  73. 73. val sum = system.actorOf(Props[Sum], "sum") val websocket = Flow[Message] .via(collectString) .map(Sample.parse) .map(sample => sum ! Increment(sample)) .mapConcat(_ => Nil)
  74. 74. val sum = system.actorOf(Props[Sum], "sum") val websocket = Flow[Message] .via(collectString) .map(Sample.parse) .map(sample => sum ! Increment(sample)) .mapConcat(_ => Nil)
  75. 75. class Sum extends Actor { var sum: Long = 0 def receive = { case Increment(value) => sum = sum + value case Query => CurrentSum(sum) } }
  76. 76. class Sum extends Actor { var sum: Long = 0 def receive = { case Increment(value) => sum = sum + value sender() ! Done case Query => CurrentSum(sum) } }
  77. 77. val websocket = Flow[Message] .via(collectString) .map(Sample.parse) .map(sample => sum ! Increment(sample)) .mapConcat(_ => Nil)
  78. 78. val websocket = Flow[Message] .via(collectString) .map(Sample.parse) .mapAsync(1) { sample => (sum ? Increment(sample)).mapTo[Done] } .mapConcat(_ => Nil)
  79. 79. ACTOR AS A SINK
  80. 80. class Sum extends Actor { var sum: Long = 0 def receive = { case Init => sender() ! Ack case Increment(value) => sum = sum + value sender() ! Ack case Query => CurrentSum(sum) } }
  81. 81. class Sum extends Actor { var sum: Long = 0 def receive = { case Init => sender() ! Ack case Increment(value) => sum = sum + value sender() ! Ack case Query => CurrentSum(sum) } }
  82. 82. class Sum extends Actor { var sum: Long = 0 def receive = { case Init => sender() ! Ack case Increment(value) => sum = sum + value sender() ! Ack case Query => CurrentSum(sum) } }
  83. 83. class Sum extends Actor { var sum: Long = 0 def receive = { case Init => sender() ! Ack case Increment(value) => sum = sum + value sender() ! Ack case Query => CurrentSum(sum) } }
  84. 84. val sum = system.actorOf(Props[Sum], "sum") val sink = Sink.actorRefWithAck[Increment](sum, Init, Ack, Complete) val websocket = Flow[Message] .via(collectString) .map(Sample.parse) .map(sample => Increment(sample)) .alsoTo(sink) .mapConcat(_ => Nil)
  85. 85. val sum = system.actorOf(Props[Sum], "sum") val sink = Sink.actorRefWithAck[Increment](sum, Init, Ack, Complete) val websocket = Flow[Message] .via(collectString) .map(Sample.parse) .map(sample => Increment(sample)) .alsoTo(sink) .mapConcat(_ => Nil)
  86. 86. val sum = system.actorOf(Props[Sum], "sum") val sink = Sink.actorRefWithAck[Increment](sum, Init, Ack, Complete) val websocket = Flow[Message] .via(collectString) .map(Sample.parse) .map(sample => Increment(sample)) .alsoTo(sink) .mapConcat(_ => Nil)
  87. 87. SENDING MESSAGES TO A STREAM
  88. 88. val consumer = Consumer.committableSource( consumerSettings, Subscriptions.topics("topic1"))
  89. 89. source .map(_.toString) .map { elem => new ProducerRecord[Array[Byte], String]("topic1", elem) } .runWith(Producer.plainSink( producerSettings))
  90. 90. Source.actorRef
  91. 91. val ref = Source.actorRef[Long](100, OverflowStrategy.fail) .to(Sink.foreach(println)) .run() Source(1 to Int.MaxValue) .map(x => ref ! x) .runWith(Sink.ignore)
  92. 92. val ref = Source.actorRef[Long](100, OverflowStrategy.fail) .to(Sink.foreach(println)) .run() Source(1 to Int.MaxValue) .map(x => ref ! x) .runWith(Sink.ignore)
  93. 93. Source.queue
  94. 94. val queue = Source.queue[Long](100, OverflowStrategy.backpressure) .to(Sink.foreach(println)) .run() Source(1 to Int.MaxValue) .mapAsync(1)(x => queue.offer(x)) .runWith(Sink.ignore)
  95. 95. val queue = Source.queue[Long](100, OverflowStrategy.backpressure) .to(Sink.foreach(println)) .run() Source(1 to Int.MaxValue) .mapAsync(1)(x => queue.offer(x)) .runWith(Sink.ignore)
  96. 96. ENCAPSULATING STREAMS WITH ACTORS
  97. 97. class PrintNumbers extends Actor { private implicit val mat = ActorMaterializer() private implicit val ec = context.system.dispatcher val stream = Source(1 to 5) .map(_.toString) .runForeach(println) .map(_ => self ! StreamComplete) def receive = { case StreamComplete => println("Stream Completed") context.stop(self) } }
  98. 98. class PrintNumbers extends Actor { private implicit val mat = ActorMaterializer() private implicit val ec = context.system.dispatcher val stream = Source(1 to 5) .map(_.toString) .runForeach(println) .map(_ => self ! StreamComplete) def receive = { case StreamComplete => println("Stream Completed") context.stop(self) } }
  99. 99. class PrintNumbers extends Actor { private implicit val mat = ActorMaterializer() private implicit val ec = context.system.dispatcher val stream = Source(1 to 5) .map(_.toString) .runForeach(println) .map(_ => self ! StreamComplete) def receive = { case StreamComplete => println("Stream Completed") context.stop(self) } }
  100. 100. 1 2 3 4 5 Stream Completed
  101. 101. LIFECYCLE MANAGEMENT
  102. 102. class PrintNumbers extends Actor { private implicit val mat = ActorMaterializer() private implicit val ec = context.system.dispatcher val stream = Source.tick(1 seconds, 1 second, 1) .scan(1)(_ + _) .map(_.toString) .runForeach(println) .map(_ => self ! StreamComplete) context.setReceiveTimeout(5 seconds) def receive = { case StreamComplete => println("Stream Completed") context.stop(self) case ReceiveTimeout => println("Receive Timeout") context.stop(self) } }
  103. 103. class PrintNumbers extends Actor { private implicit val mat = ActorMaterializer() private implicit val ec = context.system.dispatcher val stream = Source.tick(1 seconds, 1 second, 1) .scan(1)(_ + _) .map(_.toString) .runForeach(println) .map(_ => self ! StreamComplete) context.setReceiveTimeout(5 seconds) def receive = { case StreamComplete => println("Stream Completed") context.stop(self) case ReceiveTimeout => println("Receive Timeout") context.stop(self) } }
  104. 104. class PrintNumbers extends Actor { private implicit val mat = ActorMaterializer() private implicit val ec = context.system.dispatcher val stream = Source.tick(1 seconds, 1 second, 1) .scan(1)(_ + _) .map(_.toString) .runForeach(println) .map(_ => self ! StreamComplete) context.setReceiveTimeout(5 seconds) def receive = { case StreamComplete => println("Stream Completed") context.stop(self) case ReceiveTimeout => println("Receive Timeout") context.stop(self) } }
  105. 105. 1 2 3 4 5 Receive Timeout
  106. 106. class PrintNumbers extends Actor { private implicit val mat = ActorMaterializer() private implicit val ec = context.system.dispatcher val stream = Source.tick(1 seconds, 1 second, 1) .scan(1)(_ + _) .map(_.toString) .runForeach(println) .map(_ => self ! StreamComplete) context.setReceiveTimeout(5 seconds) def receive = { case StreamComplete => println("Stream Completed") context.stop(self) case ReceiveTimeout => println("Receive Timeout") context.stop(self) } }
  107. 107. class PrintNumbers(implicit mat: ActorMaterializer, ec: ExecutionContext) extends Actor { val stream = Source.tick(1 seconds, 1 second, 1) .scan(1)(_ + _) .map(_.toString) .runForeach(println) .map(_ => self ! StreamComplete) context.setReceiveTimeout(5 seconds) def receive = { case StreamComplete => println("Stream Completed") context.stop(self) case ReceiveTimeout => println("Receive timeout") context.stop(self) } }
  108. 108. 1 2 3 4 5 Receive Timeout 6 7 8 9 10 ...
  109. 109. class PrintNumbers(implicit mat: ActorMaterializer, ec: ExecutionContext) extends Actor { private val (killSwitch, done) = Source.tick(1 seconds, 1 second, 1) .scan(1)(_ + _) .map(_.toString) .viaMat(KillSwitches.single)(Keep.right) .toMat(Sink.foreach(println))(Keep.both) .run() done.map(_ => self ! StreamComplete) context.setReceiveTimeout(5 seconds) def receive = { case StreamComplete => println("Stream Completed") context.stop(self) case ReceiveTimeout => println("Receive Timeout") killSwitch.shutdown() } }
  110. 110. class PrintNumbers(implicit mat: ActorMaterializer, ec: ExecutionContext) extends Actor { private val (killSwitch, done) = Source.tick(1 seconds, 1 second, 1) .scan(1)(_ + _) .map(_.toString) .viaMat(KillSwitches.single)(Keep.right) .toMat(Sink.foreach(println))(Keep.both) .run() done.map(_ => self ! StreamComplete) context.setReceiveTimeout(5 seconds) def receive = { case StreamComplete => println("Stream Completed") context.stop(self) case ReceiveTimeout => println("Receive Timeout") killSwitch.shutdown() } }
  111. 111. 1 2 3 4 5 Receive Timeout Stream Completed
  112. 112. STREAM MANAGEMENT
  113. 113. val webSocket: Flow[Message, Message, Future[WebSocketUpgradeResponse]] = { Http().webSocketClientFlow(WebSocketRequest(websocketUri)) }
  114. 114. { "id": "c75cb448-df0e-4692-8e06-0321b7703992", "timestamp": 1486925114, "measurements": { "power": 1.7, "rotor_speed": 3.9, "wind_speed": 10.1 } }
  115. 115. val ((upgradeResponse, killSwitch), closed) = Source.fromGraph(outgoing) .viaMat(webSocket)(Keep.right) // keep Future[WebSocketUpgradeResponse] .via(incoming) .viaMat(KillSwitches.single)(Keep.both) // also keep KillSwitch .toMat(Sink.ignore)(Keep.both) // also keep Future[Done] .run()
  116. 116. val ((upgradeResponse, killSwitch), closed) = Source.fromGraph(outgoing) .viaMat(webSocket)(Keep.right) // keep Future[WebSocketUpgradeResponse] .via(incoming) .viaMat(KillSwitches.single)(Keep.both) // also keep KillSwitch .toMat(Sink.ignore)(Keep.both) // also keep Future[Done] .run()
  117. 117. val ((upgradeResponse, killSwitch), closed) = Source.fromGraph(outgoing) .viaMat(webSocket)(Keep.right) // keep Future[WebSocketUpgradeResponse] .via(incoming) .viaMat(KillSwitches.single)(Keep.both) // also keep KillSwitch .toMat(Sink.ignore)(Keep.both) // also keep Future[Done] .run()
  118. 118. val ((upgradeResponse, killSwitch), closed) = Source.fromGraph(outgoing) .viaMat(webSocket)(Keep.right) // keep Future[WebSocketUpgradeResponse] .via(incoming) .viaMat(KillSwitches.single)(Keep.both) // also keep KillSwitch .toMat(Sink.ignore)(Keep.both) // also keep Future[Done] .run()
  119. 119. val connected = upgradeResponse.map { upgrade => upgrade.response.status match { case StatusCodes.SwitchingProtocols => supervisor ! Upgraded case statusCode => supervisor ! FailedUpgrade(statusCode) } } connected.onComplete { case Success(_) => supervisor ! Connected case Failure(ex) => supervisor ! ConnectionFailure(ex) }
  120. 120. class WindTurbineSimulator(id: String, endpoint: String)(implicit mat: ActorMaterializer) extends Actor with ActorLogging { implicit val system = context.system implicit val ec = system.dispatcher val webSocket = WebSocketClient(id, endpoint, self) def receive = { case Upgraded => log.info(s"$id : WebSocket upgraded") case FailedUpgrade(statusCode) => log.error(s"$id : Failed to upgrade WebSocket connection : $statusCode") throw WindTurbineSimulatorException(id) case ConnectionFailure(ex) => log.error(s"$id : Failed to establish WebSocket connection $ex") throw WindTurbineSimulatorException(id) case Connected => log.info(s"$id : WebSocket connected") context.become(running) } def running: Receive = { case Terminated(ex) => log.error(s"$id : WebSocket connection terminated : ${ex.getMessage}") throw WindTurbineSimulatorException(id) case Completed => log.error(s"$id : WebSocket connection completed") throw WindTurbineSimulatorException(id) } override def postStop() = { log.info(s"$id : Stopping WebSocket connection") webSocket.killSwitch.shutdown() } }
  121. 121. class WindTurbineSimulator(id: String, endpoint: String)(implicit mat: ActorMaterializer) extends Actor with ActorLogging { implicit val system = context.system implicit val ec = system.dispatcher val webSocket = WebSocketClient(id, endpoint, self) def receive = { case Upgraded => log.info(s"$id : WebSocket upgraded") case FailedUpgrade(statusCode) => log.error(s"$id : Failed to upgrade WebSocket connection : $statusCode") throw WindTurbineSimulatorException(id) case ConnectionFailure(ex) => log.error(s"$id : Failed to establish WebSocket connection $ex") throw WindTurbineSimulatorException(id) case Connected => log.info(s"$id : WebSocket connected") context.become(running) } def running: Receive = { case Terminated(ex) => log.error(s"$id : WebSocket connection terminated : ${ex.getMessage}") throw WindTurbineSimulatorException(id) case Completed => log.error(s"$id : WebSocket connection completed") throw WindTurbineSimulatorException(id) } override def postStop() = { log.info(s"$id : Stopping WebSocket connection") webSocket.killSwitch.shutdown() } }
  122. 122. class WindTurbineSimulator(id: String, endpoint: String)(implicit mat: ActorMaterializer) extends Actor with ActorLogging { implicit val system = context.system implicit val ec = system.dispatcher val webSocket = WebSocketClient(id, endpoint, self) def receive = { case Upgraded => log.info(s"$id : WebSocket upgraded") case FailedUpgrade(statusCode) => log.error(s"$id : Failed to upgrade WebSocket connection : $statusCode") throw WindTurbineSimulatorException(id) case ConnectionFailure(ex) => log.error(s"$id : Failed to establish WebSocket connection $ex") throw WindTurbineSimulatorException(id) case Connected => log.info(s"$id : WebSocket connected") context.become(running) } def running: Receive = { case Terminated(ex) => log.error(s"$id : WebSocket connection terminated : ${ex.getMessage}") throw WindTurbineSimulatorException(id) case Completed => log.error(s"$id : WebSocket connection completed") throw WindTurbineSimulatorException(id) } override def postStop() = { log.info(s"$id : Stopping WebSocket connection") webSocket.killSwitch.shutdown() } }
  123. 123. class WindTurbineSimulator(id: String, endpoint: String)(implicit mat: ActorMaterializer) extends Actor with ActorLogging { implicit val system = context.system implicit val ec = system.dispatcher val webSocket = WebSocketClient(id, endpoint, self) def receive = { case Upgraded => log.info(s"$id : WebSocket upgraded") case FailedUpgrade(statusCode) => log.error(s"$id : Failed to upgrade WebSocket connection : $statusCode") throw WindTurbineSimulatorException(id) case ConnectionFailure(ex) => log.error(s"$id : Failed to establish WebSocket connection $ex") throw WindTurbineSimulatorException(id) case Connected => log.info(s"$id : WebSocket connected") context.become(running) } def running: Receive = { case Terminated(ex) => log.error(s"$id : WebSocket connection terminated : ${ex.getMessage}") throw WindTurbineSimulatorException(id) case Completed => log.error(s"$id : WebSocket connection completed") throw WindTurbineSimulatorException(id) } override def postStop() = { log.info(s"$id : Stopping WebSocket connection") webSocket.killSwitch.shutdown() } }
  124. 124. class WindTurbineSimulator(id: String, endpoint: String)(implicit mat: ActorMaterializer) extends Actor with ActorLogging { implicit val system = context.system implicit val ec = system.dispatcher val webSocket = WebSocketClient(id, endpoint, self) def receive = { case Upgraded => log.info(s"$id : WebSocket upgraded") case FailedUpgrade(statusCode) => log.error(s"$id : Failed to upgrade WebSocket connection : $statusCode") throw WindTurbineSimulatorException(id) case ConnectionFailure(ex) => log.error(s"$id : Failed to establish WebSocket connection $ex") throw WindTurbineSimulatorException(id) case Connected => log.info(s"$id : WebSocket connected") context.become(running) } def running: Receive = { case Terminated(ex) => log.error(s"$id : WebSocket connection terminated : ${ex.getMessage}") throw WindTurbineSimulatorException(id) case Completed => log.error(s"$id : WebSocket connection completed") throw WindTurbineSimulatorException(id) } override def postStop() = { log.info(s"$id : Stopping WebSocket connection") webSocket.killSwitch.shutdown() } }
  125. 125. for (_ <- 1 to 1000) { val id = java.util.UUID.randomUUID.toString system.actorOf(WindTurbineSimulator.props(id, endpoint), id) }
  126. 126. for (_ <- 1 to 1000) { Thread.sleep(13) // I never know how long to sleep :-( val id = java.util.UUID.randomUUID.toString system.actorOf(WindTurbineSimulator.props(id, endpoint), id) }
  127. 127. Source(1 to 1000) .throttle(elements = 100, per = 1 second, maximumBurst = 100, mode = ThrottleMode.shaping) .map { _ => val id = java.util.UUID.randomUUID.toString system.actorOf(WindTurbineSimulator.props(id, endpoint), id) } .runWith(Sink.ignore)
  128. 128. Source(1 to 1000) .throttle(elements = 100, per = 1 second, maximumBurst = 100, mode = ThrottleMode.shaping) .map { _ => val id = java.util.UUID.randomUUID.toString system.actorOf(WindTurbineSimulator.props(id, endpoint), id) } .runWith(Sink.ignore)
  129. 129. AKKA STREAMS SUPERVISION
  130. 130. val decider: Supervision.Decider = { case MessageParsingException(message) => log.warning(s"Unable to parse message : $message") Supervision.Resume case ex => log.error(s"Unexpected exception : $ex") Supervision.Stop }
  131. 131. val decider: Supervision.Decider = { case MessageParsingException(message) => log.warning(s"Unable to parse message : $message") Supervision.Resume case ex => log.error(s"Unexpected exception : $ex") Supervision.Stop }
  132. 132. val decider: Supervision.Decider = { case MessageParsingException(message) => log.warning(s"Unable to parse message : $message") Supervision.Resume case ex => log.error(s"Unexpected exception : $ex") Supervision.Stop }
  133. 133. BACKOFF?
  134. 134. Source(1 to 1000) .throttle(elements = 100, per = 1 second, maximumBurst = 100, mode = ThrottleMode.shaping) .map { _ => val id = java.util.UUID.randomUUID.toString system.actorOf(WindTurbineSimulator.props(id, endpoint), id) } .runWith(Sink.ignore)
  135. 135. Source(1 to 1000) .throttle(elements = 100, per = 1 second, maximumBurst = 100, mode = ThrottleMode.shaping) .map { _ => val id = java.util.UUID.randomUUID.toString val supervisor = BackoffSupervisor.props( Backoff.onFailure( WindTurbineSimulator.props(id, endpoint), childName = id, minBackoff = 1 second, maxBackoff = 30 seconds, randomFactor = 0.2 )) system.actorOf(supervisor, name = s"$id-backoff-supervisor") } .runWith(Sink.ignore)
  136. 136. DISTRIBUTED STREAM PROCESSING
  137. 137. AKKA CLUSTER SHARDING
  138. 138. class WindTurbineSupervisor(implicit mat: ActorMaterializer) extends Actor { override def preStart(): Unit = { self ! StartClient(self.path.name) } def receive = { case StartClient(id) => val supervisor = BackoffSupervisor.props( Backoff.onFailure( WindTurbineSimulator.props(id, Config.endpoint), childName = id, minBackoff = 1.second, maxBackoff = 30.seconds, randomFactor = 0.2 ) ) context.actorOf(supervisor, name = s"$id-backoff-supervisor") context.become(running) } def running(child: ActorRef): Receive = { case message => child forward message } }
  139. 139. class WindTurbineSupervisor(implicit mat: ActorMaterializer) extends Actor { override def preStart(): Unit = { self ! StartClient(self.path.name) } def receive = { case StartClient(id) => val supervisor = BackoffSupervisor.props( Backoff.onFailure( WindTurbineSimulator.props(id, Config.endpoint), childName = id, minBackoff = 1.second, maxBackoff = 30.seconds, randomFactor = 0.2 ) ) context.actorOf(supervisor, name = s"$id-backoff-supervisor") context.become(running) } def running(child: ActorRef): Receive = { case message => child forward message } }
  140. 140. class WindTurbineSupervisor(implicit mat: ActorMaterializer) extends Actor { override def preStart(): Unit = { self ! StartClient(self.path.name) } def receive = { case StartClient(id) => val supervisor = BackoffSupervisor.props( Backoff.onFailure( WindTurbineSimulator.props(id, Config.endpoint), childName = id, minBackoff = 1.second, maxBackoff = 30.seconds, randomFactor = 0.2 ) ) context.actorOf(supervisor, name = s"$id-backoff-supervisor") context.become(running) } def running(child: ActorRef): Receive = { case message => child forward message } }
  141. 141. class WindTurbineSupervisor(implicit mat: ActorMaterializer) extends Actor { override def preStart(): Unit = { self ! StartClient(self.path.name) } def receive = { case StartClient(id) => val supervisor = BackoffSupervisor.props( Backoff.onFailure( WindTurbineSimulator.props(id, Config.endpoint), childName = id, minBackoff = 1.second, maxBackoff = 30.seconds, randomFactor = 0.2 ) ) context.actorOf(supervisor, name = s"$id-backoff-supervisor") context.become(running) } def running(child: ActorRef): Receive = { case message => child forward message } }
  142. 142. val shardRegion = ClusterSharding(system).start( typeName = "WindTurbineSupervisorShardRegion", entityProps = WindTurbineSupervisor.props, settings = ClusterShardingSettings(system), extractEntityId = WindTurbineClusterConfig.extractEntityId, extractShardId = WindTurbineClusterConfig.extractShardId)
  143. 143. val shardRegion = ClusterSharding(system).start( typeName = "WindTurbineSupervisorShardRegion", entityProps = WindTurbineSupervisor.props, settings = ClusterShardingSettings(system), extractEntityId = WindTurbineClusterConfig.extractEntityId, extractShardId = WindTurbineClusterConfig.extractShardId)
  144. 144. val shardRegion = ClusterSharding(system).start( typeName = "WindTurbineSupervisorShardRegion", entityProps = WindTurbineSupervisor.props, settings = ClusterShardingSettings(system), extractEntityId = WindTurbineClusterConfig.extractEntityId, extractShardId = WindTurbineClusterConfig.extractShardId)
  145. 145. val shardRegion = ClusterSharding(system).start( typeName = "WindTurbineSupervisorShardRegion", entityProps = WindTurbineSupervisor.props, settings = ClusterShardingSettings(system), extractEntityId = WindTurbineClusterConfig.extractEntityId, extractShardId = WindTurbineClusterConfig.extractShardId)
  146. 146. Source(1 to 100000) .throttle(elements = 100, per = 1 second, maximumBurst = 100, mode = ThrottleMode.shaping) .map { _ => val id = java.util.UUID.randomUUID.toString shardRegion ! EntityEnvelope(id, StartSimulator) } .runWith(Sink.ignore)
  147. 147. LOCATION TRANSPARENCY
  148. 148. class WindTurbineSimulator(id: String, endpoint: String) (implicit mat: ActorMaterializer) extends Actor with ActorLogging { implicit val system = context.system implicit val ec = system.dispatcher val webSocket = WebSocketClient(id, endpoint, self) def receive = { case Upgraded => log.info(s"$id : WebSocket upgraded") case FailedUpgrade(statusCode) => log.error(s"$id : Failed to upgrade WebSocket connection : $statusCode") throw WindTurbineSimulatorException(id) case ConnectionFailure(ex) => log.error(s"$id : Failed to establish WebSocket connection $ex") throw WindTurbineSimulatorException(id) case Connected => log.info(s"$id : WebSocket connected") context.become(running) } def running: Receive = { case Terminated(ex) => log.error(s"$id : WebSocket connection terminated : ${ex.getMessage}") throw WindTurbineSimulatorException(id) case Completed => log.error(s"$id : WebSocket connection completed") throw WindTurbineSimulatorException(id) } override def postStop() = { log.info(s"$id : Stopping WebSocket connection") webSocket.killSwitch.shutdown() } }
  149. 149. def running: Receive = { case QueryActorPath => sender() ! SimulatorActorPath(context.self.path.toString) case Terminated(ex) => log.error(s"$id : WebSocket connection terminated : ${ex.getMessage}") throw WindTurbineSimulatorException(id) case Completed => log.error(s"$id : WebSocket connection completed") throw WindTurbineSimulatorException(id) }
  150. 150. val id = "6c84fb90-12c4-11e1-840d-7b25c5ee775a" (shardRegion ? EntityEnvelope(id, QueryActorPath)) .mapTo[SimulatorActorPath]
  151. 151. val id = "6c84fb90-12c4-11e1-840d-7b25c5ee775a" (shardRegion ? EntityEnvelope(id, QueryActorPath)) .mapTo[SimulatorActorPath]
  152. 152. val route = path(uuidRegex) { id => get { val query = (shardRegion ? EntityEnvelope(id, QueryActorPath)) .mapTo[SimulatorActorPath] onSuccess(query) { response => complete(response) } } }
  153. 153. BLURRING THE LINES
  154. 154. PERSISTING STREAMS WITH ACTORS
  155. 155. AKKA PERSISTENCE
  156. 156. val websocket = (diagnosticEvent: ActorRef) => Flow[Message] .via(collectString) .mapAsync(4) { event => diagnosticEvent ? Event(event) } .mapConcat(_ => Nil)
  157. 157. val route = path(uuidRegex) { id => get { val ref = system.actorOf(DiagnosticEvent.props(id), id) handleWebSocketMessages { websocket(ref) .watchTermination() { (_, done) => done.map(_ => ref ! Stop) } } } }
  158. 158. class DiagnosticEvent(id: String) extends PersistentActor { override def persistenceId = id override def receiveRecover = Actor.emptyBehavior override def receiveCommand = { case event: Event => persist(event) { event => sender() ! Done } case Stop => context.stop(self) } }
  159. 159. class DiagnosticEvent(id: String) extends PersistentActor { override def persistenceId = id override def receiveRecover = Actor.emptyBehavior override def receiveCommand = { case event: Event => persist(event) { event => sender() ! Done } case Stop => context.stop(self) } }
  160. 160. class DiagnosticEvent(id: String) extends PersistentActor { override def persistenceId = id override def receiveRecover = Actor.emptyBehavior override def receiveCommand = { case event: Event => persist(event) { event => sender() ! Done } case Stop => context.stop(self) } }
  161. 161. STREAMING PERSISTED EVENTS
  162. 162. AKKA PERSISTENCE QUERY
  163. 163. val id = "123e4567-e89b-12d3-a456-426655440000" queries.currentEventsByPersistenceId(id, 0L, Long.MaxValue) .map(_.event) .runWith(Sink.foreach(println))
  164. 164. val id = "123e4567-e89b-12d3-a456-426655440000" queries.currentEventsByPersistenceId(id, 0L, Long.MaxValue) .map(_.event) .runWith(Sink.foreach(println))
  165. 165. val route = path("events" / uuidRegex) { id => onSuccess { queries.currentEventsByPersistenceId(id, 0L, Long.MaxValue) .map(_.event) .runWith(Sink.seq) .mapTo[Seq[Event]] } { events => complete(events) } }
  166. 166. val route = path("events" / uuidRegex) { id => get { parameters('bookmark.as[Long] ? 0L) { bookmark => onSuccess { queries.currentEventsByPersistenceId(id, bookmark, bookmark + 1000) .runWith(Sink.seq) .mapTo[Seq[EventEnvelope]] } { envelopes => if (envelopes.isEmpty) { complete(StatusCodes.NoContent) } else { val bookmark = envelopes.last.sequenceNr + 1 val events = envelopes.map(_.event.asInstanceOf[Event]) complete(EventsWithBookmark(bookmark, events)) } } } } }
  167. 167. val id = "123e4567-e89b-12d3-a456-426655440000" queries.eventsByPersistenceId(id, 0L, Long.MaxValue) .map(_.event) .runWith(Sink.foreach(println))
  168. 168. val websocket = Flow[Message] .collect { case TextMessage.Streamed(textStream) => textStream.runWith(Sink.ignore) Nil } .merge(queries.eventsByPersistenceId(id, 0L, Long.MaxValue).map(_.event), eagerComplete = true) .map(event => TextMessage(event.toString))
  169. 169. LOCATION TRANSPARENCY
  170. 170. SUMMARY
  171. 171. ACTORS> Managing State > Lifecycle Management > Concurrency Model > Workload Distribution > Queryable > Location Transparency > Digital Twin
  172. 172. STREAMS > Streaming Workloads > Backpressure > Bounded Resource Constraints > System Dynamics > Powerful Streaming Semantics > Rethink Workloads
  173. 173. ACTORS AND STREAMS > Fast data and complex state and location transparency > Telemetry and asset metadata and relationships > Composable and Extensible > and integration (HTTP, Alpakka) > Reactive systems and reactive applications
  174. 174. @breckcs http://blog.colinbreck.com/ 1. AKKA STREAMS: A MOTIVATING EXAMPLE 2. PATTERNS FOR STREAMING MEASUREMENT DATA WITH AKKA STREAMS 3. INTEGRATING AKKA STREAMS AND AKKA ACTORS (PART I-IV)
  175. 175. QUESTIONS?

×