Rising Above_ Dubai Floods and the Fortitude of Dubai International Airport.pdf
The dark side of Akka and the remedy
1. The dark side of Akka
and the remedy
V2
My experiences with Akka
Ákos Kriváchy
2. Introduction - Ákos Kriváchy
• Scala fanatic since 2013 February
• First FP language
• Akka since 2014 February
• Things I love about Scala:
• Static type system
• Type classes
• Partial Functions
• foldLeft, tail recursion, functional features
2
3. Today’s topic
1. Overview of Akka
2. When to use Akka
3. Some caveats
4. Possible solutions
5. Conclusions
6. Actors
• Actor Model
• Lightweight event-driven processes
• 2.7 million actors per GB RAM
• Communication only via messages
• Message processing guaranteed to be on a single thread
6
Source: http://doc.akka.io/docs/akka/snapshot/intro/what-is-akka.html
8. Actor implementation
class MyActor extends Actor {
def receive: Receive = {
case Message(data) =>
// ...
case OtherMessage(_) =>
// ...
context.become(otherBehavior)
}
def otherBehavior: Receive = …
}
Behavior:
PartialFunction[Any, Unit]
Approximately in Java:
PartialFunction[Object,
Able to change behavior
9. Fault Tolerance
• Supervisor hierarchies
• “Let-it-crash" semantics.
• Location transparency
• Different JVMs on different servers on different continents
• Highly fault-tolerant systems
• self-heal and never stop
• Persistence and recovery
• Journaling of all messages
Source: http://doc.akka.io/docs/akka/snapshot/intro/what-is-akka.html
12. •Akka solves for us:
• Parallelism
• Scalability and distributability
• Resilience
•Perfect for:
• Transaction/(semi-)real-time processing
• Backend with high parallelism
• Calculations
• High availability backends
To Akka or not to Akka?
14. Thinking in Actors
Number of Actors
Only few stateful
components are Actors
Everything is an Actor
Super-complicated
logic inside Actors
Complexity in
communication
(“infrastructure”)
15. Thinking in Actors
Number of Actors
Only few stateful
components are Actors
Everything is an Actor
Monoliths Micro-services
17. Problem #1: Any and Actor Ref
• All messages are Any-s
• Anything that’s not
handled ends up as a
“dead letter”
• Requires extensive
testing to “feel safe”
class MyActor(databasePersistence: ActorRef,
emailSender : ActorRef,
MQSender : ActorRef,
widgetDao : ActorRef,
twitterService : ActorRef)
extends Actor {
def receive: Receive = {
case Message(data) =>
twitterService ! Tweet(data)
// ...
case OtherMessage(_) =>
// ...
}
}
}
17
18. def receive: Receive = {
case Message(data) =>
emailSender ! Tweet(data)
// ...
case OtherMessage(_) =>
// ...
}
22. Joking aside:
Theory around static type-checking
state machines is hard
Differing opinions on how severe the issue is (cost vs. benefit):
http://stew.vireo.org/posts/I-hate-akka/
https://www.reddit.com/r/scala/comments/2ruskl/i_hate_akka/
23. Solution coming: Akka Typed
• Unreleased and experimental
• Defined protocols with types for:
• ActorSystem!
• No user guardian, define your own!
• Actor
• ActorRef
• .tell (!)
• .ask (?)
• More information:
• http://doc.akka.io/docs/akka/snapshot/scala/typed.html
• http://blog.scalac.io/2015/04/30/leszek-akka-typed.html
24. In the meantime: Don’t use Actors for everything
• IMHO: Benefits outweigh the cost of the loss of type safety in
certain applications
• What do we use instead?
• Scala Futures are extremely powerful
• Futures compose nicer than Actors
• What about supervision and „let it crash”?
• Future’s handle failure also
24
26. var hell
• Actors have too much mutable state
• Our worst scenario: Actor with 300 lines and 20 vars
• Hard to reason about state when everything is “global” inside
the Actor
• How do you initialize state that is only used in some
behaviours?
• var something: SomeType = _
• NPE
• var something: Option[SomeType] = None
• Always need to “getOrElse”
26
27. become/unbecome hell
• Pushing and popping state on a stack
• context.become(behavior: Receive, discardOld: Boolean = true)
• context.unbecome()
• “context.become” isn’t enforced to be called last
• You use methods to keep things short, but there will be multiple
methods trying to modify the behaviour
• i.e. you could end up inadvertently overwriting behavior
• One place: context.become(handleCoolMessages orElse waitForNewRequests)
• Somewhere else: context.become(waitForNewRequests, discardOld = true)
• When things blow up you have no idea how you got there
27
31. Debug hell
• Debugging Actors is hard
• Stacktraces lack meaning
• Need to have a lot of boilerplate utility code:
• What was the message?
• Who sent the message?
• What’s my internal state?
31
32. [ERROR] [akka://simple-actor/user/receptionist/controller-1/$a] Exception happened
java.lang.Exception: exception happened here
at meetup.akka.simple.Getter$$anonfun$receive$1$$anonfun$1.apply(Getter.scala:37)
at meetup.akka.simple.Getter$$anonfun$receive$1$$anonfun$1.apply(Getter.scala:37)
at scala.util.Try$.apply(Try.scala:191)
at meetup.akka.simple.Getter$$anonfun$receive$1.applyOrElse(Getter.scala:37)
at akka.actor.Actor$class.aroundReceive(Actor.scala:465)
at meetup.akka.simple.Getter.aroundReceive(Getter.scala:16)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)
at akka.actor.ActorCell.invoke(ActorCell.scala:487)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238)
at akka.dispatch.Mailbox.run(Mailbox.scala:220)
at akka.dispatch.ForkJoinExecutorConfigurator
$AkkaForkJoinTask.exec(AbstractDispatcher.scala:393)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:
107) 32
33. We need Logging for Debugging
• Akka provides:
• with ActorLogging
• log.info(…)
• LoggingRecieve: def receive = LoggingReceive { … }
• Lifecycle logging: akka.actor.debug.lifecycle=true
• Autorecieve logging: akka.actor.debug.autoreceive=true
• Issue:
• Akka only provides logging of actual messages per Receive block (has
pros and cons)
• If you missed one => good luck debugging issues around it in
production
33
34. [DEBUG] [akka://simple-actor/user/receptionist] started(meetup.akka.simple.Receptionist@14ba772)
[DEBUG] [akka://simple-actor/user/receptionist/controller-1] started
(meetup.akka.simple.Controller@38a5d7)
[DEBUG] [akka://simple-actor/user/receptionist] now supervising Actor[akka://simple-
actor/user/receptionist/controller-1#4232237]
[DEBUG] [akka://simple-actor/user/receptionist/controller-1] now watched by
Actor[akka://simple-actor/user/receptionist#1565954732]
[DEBUG] [akka://simple-actor/user/receptionist/controller-1] received handled
message Check(http://doc.akka.io/docs/akka/2.3.5/intro/what-is-akka.html,1)
Logging example
34
40. FSM concepts
object Receptionist {
object Internal {
sealed trait State
case object Sleeping extends State
case object Processing extends State
sealed trait Data
case class NoQueue(requestId: Int = 0) extends Data
case class Queue(currentRequestId: Int, items: Vector[Job]) extends Data
}
}
40
States explicitly
defined – just typed
singletons
Data used in states
explicitly defined
42. Define handlers for all states
// Initialize with data
startWith(Sleeping, NoQueue())
// Handlers for states
when(Sleeping)(enqueueNewRequest)
when(Processing) (
processResult
orElse enqueueNewRequest
orElse reportError
)
42
Isn’t this
nice?
44. def enqueueNewRequest: StateFunction = {
case Event(Api.Scrape(url, depth), NoQueue(requestId)) =>
// …
case Event(Api.Scrape(url, depth), queue: Queue) =>
if (queue.items.size > 3) {
stay replying Api.Failed(url)
} else {
goto(Processing) using
Queue(queue.currentRequestId, queue.items :+ Job(sender(), url,
depth))
}
}
Message Data in state
State changes
Data changes
45. Monitoring state transitions
onTransition {
case Idle -> Active => setTimer("timeout", Tick, 1 second, true)
case Active -> _ => cancelTimer("timeout")
case x -> Idle => log.info("entering Idle from " + x)
}
monitoredActor ! SubscribeTransitionCallBack(self)
def recieve = {
case Transition(monitoredActor, oldState, newState) =>
if (newState == Errored) alert.raiseAlert(...)
}
override def postStop() = {
monitoredActor ! UnsubscribeTransitionCallBack(self)
}
Internal:
External:
45
46. Handling failure
whenUnhandled {
case Event(any, data) =>
val logUpToHere = prettyPrint(getLog)
log.error(s"Unhandled event: ${any}n${logUpToHere}")
stay()
}
46
47. Handling failure
def handleFailureMessage: StateFunction = {
case Event(Status.Failure(cause), _) =>
log.error(s"Failed to GET $url", cause)
stop(FSM.Failure(cause))
}
onTermination {
case StopEvent(FSM.Normal, state, data) => ???
case StopEvent(FSM.Shutdown, state, data) => ???
case StopEvent(FSM.Failure(cause), state, data) => ???
}
48. Result is simplicity
class Receptionist extends FSM[Internal.State, Internal.Data] {
startWith(Sleeping, NoQueue())
when(Sleeping)(enqueueNewRequest)
when(Processing) (processResult orElse enqueueNewRequest orElse reportError)
def enqueueNewRequest: StateFunction = ???
def processResult: StateFunction = ???
def reportError: StateFunction = ???
whenUnhandled { ??? }
initialize()
}
48
Possible states
of Actor explicit
Behavior self-
contained
Error handling
common
49. Hell status
var ✔
No more mutable global state
inside Actors. Everything is
typed to the specific State.
become/unbecome ✔
All methods have to end in a
state transition. States are
clearly defined what they do.
debug/logging ?
49
51. LoggingFSM
• Remembers state transitions:
• Can override size of history (logDepth defaults to 8)
• Works super well with: onTermination
• Debug logging: akka.actor.debug.fsm=true
• Automatically logs important Events: message + internal
data
• Logs state transitions
• Use with: akka.actor.debug.lifecycle=true
51
53. [ERROR] [akka://fsm/user/receptionist] Unhandled event: some string
Last 8 entries leading up to this point:
in state: Sleeping
with data: NoQueue(2)
received: Scrape(http://non-existent.link,5)
in state: Processing
with data: Queue(3,Vector(Job(Actor[akka://fsm/system/testActor1#758674372],http://
non-existent.link,5)))
received: Result(Set(http://non-existent.link))
[…]
in state: Processing
with data: Queue(4,Vector(Job(Actor[akka://fsm/system/testActor1#758674372],http://
non.existent1,0), Job(Actor[akka://fsm/system/testActor1#758674372],http://
non.existent2,0), Job(Actor[akka://fsm/system/testActor1#758674372],http://
non.existent3,0), Job(Actor[akka://fsm/system/testActor1#758674372],http://
non.existent4,0)))
received: some string
53
Awesome
history of what
happened
54. Hell status
var ✔
No more mutable global state
inside Actors. Everything is
typed to the specific State.
become/unbecome ✔
All methods have to end in a
state transition. States are
clearly defined what they do.
debug/logging ✔
FSM does all the debug
logging we would ever need.
54
59. My experiences
•Needs a new mindset
•Unexperienced developers + learning curve +
easy to make mistakes = headaches
•Use Actors only when really needed
• Too much testing of protocols and communication
•Try to make all Actors FSM Actors
•Helps solve issues around Actor complexity
@akoskrivachy https://github.com/krivachy/AkkaWithFsm