Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

The dark side of Akka and the remedy - bp.scala meetup

7,951 views

Published on

We explore some common issues with Akka and look at how using FSM Actors can alleviate them.

Published in: Technology
  • Follow the link, new dating source: ♥♥♥ http://bit.ly/2ZDZFYj ♥♥♥
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Dating for everyone is here: ❤❤❤ http://bit.ly/2ZDZFYj ❤❤❤
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

The dark side of Akka and the remedy - bp.scala meetup

  1. 1. The dark side of Akka and the remedy Ákos Kriváchy
  2. 2. Introduction - Ákos Kriváchy • Scala disciple/convert/fanatic since 2013 February • First FP language • Akka since 2014 February • Things I love about Scala: • Static typing • Partial Functions • foldLeft, tail recursion 2
  3. 3. Recap on Akka • Akka in key points: • Messaging • Actors • Mailboxes (ActorRefs) • Hierarchy • Supervision • Location transparency Source: http://www.scottlogic.com/blog/2014/08/15/using-akka-and-scala-to-render-a-mandelbrot-set.html 3
  4. 4. Example from previous Meetup: URL scraping 4
  5. 5. To Akka or not to Akka? • Akka solves: • Concurrency • Scalability and distributability • Resilience • Akka in our team: •We had to rewrite a legacy Java component • Chose Akka to try it out (after 4 months of research) 5
  6. 6. Problem #1: Any and Actor Ref • All messages are Any-s • Anything that’s not handled ends up as a “dead letter” • Essentially a loss of all typesafety • Requires extensive testing to “feel safe” class MyActor(databasePersistence: ActorRef, emailSender: ActorRef, MQSender: ActorRef, widgetDao: ActorRef, twitterService: ActorRef) extends Actor { def receive: Receive = { case Message(data) => twitterService ! Tweet(data) // ... case OtherMessage(_) => // ... } } } 6
  7. 7. Solution: Typed Channels val channelA: ChannelRef[(MsgA, MsgB) :+: TNil] = ??? val a = new MsgA channelA <-!- a // send a to channelA a -!-> channelA // same thing as above val fB: Future[MsgB] = channelA <-?- a // ask the actor Instead of the current solution: val fB: Future[Any] = actorRef ? a http://doc.akka.io/docs/akka/2.2.0/scala/typed-channels.html 7
  8. 8. NOPE! http://doc.akka.io/docs/akka/2.3.0-RC1/project/migration-guide-2.2.x-2.3.x.html 8
  9. 9. It’s on the roadmap 9 August 28, 2014: http://typesafe.com/blog/akka-roadmap-update-2014 Lastly, we dare a look beyond the next major release: after Akka 2.4.0 [early 2015] we will tackle the number one feature request, namely more type-safety for inter-Actor messaging. This will be a major research effort and likely also a disruptive API change. For this reason Akka 2.4 is planned to be a long-term support release, because even though we will try to find ways for compatibility or at least an easy migration path we cannot currently foresee the exact nature of all related changes.
  10. 10. In the meantime: Don’t use Actors for everything • What do we use instead? • Scala Futures are extremely powerful • Futures compose more nicely than Actors • When should we use Actors? • State in the application • A cache is not state • Extreme scaling is needed (location transparency) • Need the flexibility to place any operation on any JVM -> use Actors for everything • Realtime data flow applications (not request/response) • What about supervision and „let it crash”? • Future’s handle failure also 10
  11. 11. Code: https://github.com/krivachy/AkkaWithFsm 11
  12. 12. class Getter(url: String, depth: Int) extends Actor with ActorLogging with LinkParser { lazy val client = new NingWSClient(new Builder().build()).url(url) override def preStart() = { client.get().map(_.body).map(Response).pipeTo(self) } def receive = { case Response(body) => val links = parseLinks(body).toList log.info(s"URL $url at depth $depth had ${links.size} links.") links.foreach { link => log.info(s"Sending link '$link'") context.parent ! Controller.Check(link, depth) } context.stop(self) case Status.Failure(cause) => log.error(s"Failed to GET $url", cause) context.stop(self) } } 12
  13. 13. Simplified with Futures: object Getter extends LinkParser { case class Urls(urls: Set[String], depth: Int) lazy val client = new NingWSClient(new Builder().build()) def get(url: String, depth: Int)(implicit ec: ExecutionContext): Future[Urls] = { client.url(url).get().map { response => Urls(parseLinks(response.body).toSet, depth) } } } 13
  14. 14. Problem #2: Hellish code complexity 14
  15. 15. var hell • Actors have too much mutable state • Our worst scenario: Actor with 300 lines and 20 vars • Hard to reason about state when everything is “global” inside the Actor • How do you initialize state that is only used in some cases? • var something: SomeType = _ • NPE • var something: Option[SomeType] = None • Always need to “getOrElse” 15
  16. 16. become/unbecome hell • Pushing and popping state on a stack • context.become(behavior: Receive, discardOld: Boolean = true) • context.unbecome() • “context.become” isn’t enforced to be called last • You use methods to keep things short, but there will be multiple methods trying to modify the behaviour • i.e. you could end up inadvertently overwriting behavior • One place: context.become(handleCoolMessages orElse waitForNewRequests) • Somewhere else: context.become(waitForNewRequests, discardOld = true) • When things blow up you have no idea how you got there 16
  17. 17. class Receptionist extends Actor { def receive = waiting def waiting: Receive = { case Api.Scrape(url, depth) => context.become(next(Vector(Job(sender, url, depth)))) } def running(queue: Vector[Job]): Receive = LoggingReceive { case Result(links) => // … context.become(next(queue.tail)) case Api.Scrape(url, depth) => context.become(enqueue(queue, Job(sender, url, depth))) case Terminated(_) => // … context.become(next(queue.tail)) } def enqueue(queue: Vector[Job], job: Job): Receive = LoggingReceive { // … running(queue) } def next(queue: Vector[Job]): Receive = LoggingReceive { // … running(queue) } } 17
  18. 18. Debug hell • Debugging Actors is hard • Stacktraces lack meaning • Need to have a lot of boilerplate utility code: • What was the message? • Who sent the message? • What’s my internal state? • Possible solution: • always override def preRestart(reason: Throwable, message: Option[Any]) • Still don’t know how we ended up in that become/unbecome state. 18
  19. 19. [ERROR] [akka://simple-actor/user/receptionist/controller-1/$a] Exception happened java.lang.Exception: exception happened here at meetup.akka.simple.Getter$$anonfun$receive$1$$anonfun$1.apply(Getter.scala:37) at meetup.akka.simple.Getter$$anonfun$receive$1$$anonfun$1.apply(Getter.scala:37) at scala.util.Try$.apply(Try.scala:191) at meetup.akka.simple.Getter$$anonfun$receive$1.applyOrElse(Getter.scala:37) at akka.actor.Actor$class.aroundReceive(Actor.scala:465) at meetup.akka.simple.Getter.aroundReceive(Getter.scala:16) at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516) at akka.actor.ActorCell.invoke(ActorCell.scala:487) at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238) at akka.dispatch.Mailbox.run(Mailbox.scala:220) at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:393) at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) 19
  20. 20. We need Logging for Debugging • Akka provides: • with ActorLogging • log.info(…) • LoggingRecieve: def receive = LoggingReceive { … } • turn on: akka.actor.debug.receive=true • Lifecycle logging: akka.actor.debug.lifecycle=true • For all Actors • Start, restart, stop, supervise, watch events • Autorecieve logging: akka.actor.debug.autoreceive=true • For all Actors • Automatically handled messages: Stop, Kill, PoisionPill, Terminated, etc. • Issue: • Akka only provides logging of messages per Receive block (has pros and cons) • If you missed one => good luck debugging issues around it in production 20
  21. 21. Logging example [DEBUG] [akka://simple-actor/user/receptionist] started (meetup.akka.simple.Receptionist@14ba772) [DEBUG] [akka://simple-actor/user/receptionist/controller-1] started (meetup.akka.simple.Controller@38a5d7) [DEBUG] [akka://simple-actor/user/receptionist] now supervising Actor[akka://simple-actor/user/receptionist/controller-1#4232237] [DEBUG] [akka://simple-actor/user/receptionist/controller-1] now watched by Actor[akka://simple-actor/user/receptionist#1565954732] [DEBUG] [akka://simple-actor/user/receptionist/controller-1] received handled message Check(http://doc.akka.io/docs/akka/2.3.5/intro/what-is-akka.html,1) 21
  22. 22. FSM 22
  23. 23. “Everything* is a Finite State Machine.”** -me *not everything ** do not quote me 23
  24. 24. FSM concepts object Receptionist { object Internal { sealed trait State case object Sleeping extends State case object Processing extends State sealed trait Data case class NoQueue(requestId: Int = 0) extends Data case class Queue(currentRequestId: Int, items: Vector[Job]) extends Data } } class Receptionist extends FSM[Internal.State, Internal.Data] { … } 24
  25. 25. Define handlers for all states // Initialize with data startWith(Sleeping, NoQueue()) // Handlers for states when(Sleeping)(enqueueNewRequest) when(Processing) (processResult orElse enqueueNewRequest orElse reportError) 25
  26. 26. Enqueue New Request State Function def enqueueNewRequest: StateFunction = { case Event(Api.Scrape(url, depth), NoQueue(requestId)) => startControllerFor(requestId + 1, Vector(Job(sender(), url, depth))) case Event(Api.Scrape(url, depth), queue: Queue) => if (queue.items.size > 3) { stay replying Api.Failed(url) } else { goto(Processing) using Queue(queue.currentRequestId, queue.items :+ Job(sender(), url, depth)) } } • Important: • StateFunction: Event => State • Event(incomingMessage, data) => • State transition: goto/stay (nextState) using (data) forMax(timeout) replying (message) 26
  27. 27. Monitoring state transitions Internal: onTransition { case Idle -> Active => setTimer("timeout", Tick, 1 second, true) case Active -> _ => cancelTimer("timeout") case x -> Idle => log.info("entering Idle from " + x) } External: monitoredActor ! SubscribeTransitionCallBack(self) def recieve = { case Transition(monitoredActor, oldState, newState) => if (newState == Errored) alert.raiseAlert(...) } override def postStop() = { monitoredActor ! UnsubscribeTransitionCallBack(self) } 27
  28. 28. Handling failure whenUnhandled { case Event(any, data) => val logUpToHere = prettyPrint(getLog) log.error(s"Unhandled event: ${any}n$logUpToHere") stay() } def failure: StateFunction = { case Event(Status.Failure(cause), _) => log.error(s"Failed to GET $url", cause) stop(FSM.Failure(cause)) } onTermination { case StopEvent(FSM.Normal, state, data) => ??? case StopEvent(FSM.Shutdown, state, data) => ??? case StopEvent(FSM.Failure(cause), state, data) => ??? } 28
  29. 29. Result is simplicity class Receptionist extends FSM[Internal.State, Internal.Data] { startWith(Sleeping, NoQueue()) when(Sleeping)(enqueueNewRequest) when(Processing) (processResult orElse enqueueNewRequest orElse reportError) def enqueueNewRequest: StateFunction = ??? def processResult: StateFunction = ??? def reportError: StateFunction = ??? private def nextQueueItem(queue: Queue): State = ??? private def startControllerFor(requestId: Int, queue: Vector[Job]): State = ??? whenUnhandled { ??? } initialize() } 29
  30. 30. Hell status var ✔ No more mutable global state inside Actors. Everything is typed to the specific State. become/unbecome ✔ All methods have to end in a state transition. States are clearly defined what they do. debug/logging ? 30
  31. 31. LoggingFSM • Remembers state transitions: • override def logDepth = 8 • def getLog: Seq[LogEntry[Internal.State, Internal.Data]] • Use with: onTermination • Debug logging: akka.actor.debug.fsm=true • Automatically logs important Events: message + internal data • Logs state transitions • Use with: akka.actor.debug.lifecycle=true 31
  32. 32. Debug log example [DEBUG] [akka://fsm/user/receptionist] processing Event(Result(Set([…])),Queue(1,Vector(Job(Actor[akka://fsm/system/testActor1#758674372],http://doc.akka.io /docs/akka/2.3.5/intro/what-is-akka.html,1)))) from Actor[akka://fsm/user/receptionist/controller-1#4232237] [DEBUG] [akka://fsm/user/receptionist/controller-1] transition CollectingResults -> Completed [DEBUG] [akka://fsm/user/receptionist] transition Processing -> Sleeping [DEBUG] [akka://fsm/user/receptionist/controller-1] stopped 32
  33. 33. [ERROR] [akka://fsm/user/receptionist] Unhandled event: some string Last 8 entries leading up to this point: in state: Sleeping with data: NoQueue(2) received: Scrape(http://non-existent.link,5) in state: Processing with data: Queue(3,Vector(Job(Actor[akka://fsm/system/testActor1#758674372],http://non-existent. link,5))) received: Result(Set(http://non-existent.link)) […] in state: Processing with data: Queue(4,Vector(Job(Actor[akka://fsm/system/testActor1#758674372],http://non.existent1,0), Job(Actor[akka://fsm/system/testActor1#758674372],http://non.existent2,0), Job(Actor[akka://fsm/system/testActor1#758674372],http://non.existent3,0), Job(Actor[akka://fsm/system/testActor1#758674372],http://non.existent4,0))) received: some string 33
  34. 34. Hell status var ✔ No more mutable global state inside Actors. Everything is typed to the specific State. become/unbecome ✔ All methods have to end in a state transition. States are clearly defined what they do. debug/logging ✔ FSM does all the debug logging we would ever need. 34
  35. 35. Conclusion: My experiences Some people, when confronted with a scalability problem, think “I know, I'll use Akka.” Now they have two problems. • Akka is awesome, but: • Needs a new mindset • Unexperienced developers + learning curve + easy to make mistakes = headaches • In large teams (30+) where everyone is expected to be able to change all code it becomes an issue. • Use Actors only when really needed • All Actors should probably be FSM @krivachy https://github.com/krivachy/AkkaWithFsm

×