The dark side of Akka and the remedy

The dark side of Akka 
and the remedy 
V2
My experiences with Akka 
 
Ákos Kriváchy

Introduction - Ákos Kriváchy
• Scala fanatic since 2013 February
• First FP language
• Akka since 2014 February
• Things I love about Scala:
• Static type system
• Type classes
• Partial Functions
• foldLeft, tail recursion, functional features
2

Today’s topic
1. Overview of Akka
2. When to use Akka
3. Some caveats
4. Possible solutions
5. Conclusions

Simple and high-level abstractions for
concurrency and parallelism

Actors
• Actor Model
• Lightweight event-driven processes
• 2.7 million actors per GB RAM
• Communication only via messages
• Message processing guaranteed to be on a single thread
6
Source: http://doc.akka.io/docs/akka/snapshot/intro/what-is-akka.html

Image Source: http://www.scottlogic.com/blog/2014/08/15/using-akka-and-scala-to-render-a-mandelbrot-
set.html

Actor implementation
class MyActor extends Actor {
def receive: Receive = {
case Message(data) =>
// ...
case OtherMessage(_) =>
// ...
context.become(otherBehavior)
}
def otherBehavior: Receive = …
}
Behavior:
PartialFunction[Any, Unit] 
 
Approximately in Java: 
PartialFunction[Object,
Able to change behavior

Fault Tolerance
• Supervisor hierarchies
• “Let-it-crash" semantics.
• Location transparency
• Different JVMs on different servers on different continents
• Highly fault-tolerant systems
• self-heal and never stop
• Persistence and recovery
• Journaling of all messages
Source: http://doc.akka.io/docs/akka/snapshot/intro/what-is-akka.html

Image Source: http://doc.akka.io/docs/akka/snapshot/general/supervision.html

•Akka solves for us:
• Parallelism
• Scalability and distributability
• Resilience
•Perfect for:
• Transaction/(semi-)real-time processing
• Backend with high parallelism
• Calculations
• High availability backends
To Akka or not to Akka?

Thinking in Actors
Number of Actors
Only few stateful
components are Actors
Everything is an Actor

Thinking in Actors
Number of Actors
Only few stateful
Super-complicated
logic inside Actors
Complexity in
communication 
(“infrastructure”)

Thinking in Actors
Number of Actors
Only few stateful
Monoliths Micro-services

Problem #1: Any and Actor Ref
• All messages are Any-s
• Anything that’s not
handled ends up as a
“dead letter”
• Requires extensive
testing to “feel safe”
class MyActor(databasePersistence: ActorRef,
emailSender : ActorRef,
MQSender : ActorRef,
widgetDao : ActorRef,
twitterService : ActorRef)
extends Actor {
twitterService ! Tweet(data)
// ...
// ...
}
}
}
17

 
 
emailSender ! Tweet(data)
// ...  
// ...
}

 
 
emailSender ! Tweet(data)
// ...  
// ...
}
BUG

Joking aside:
Theory around static type-checking 
state machines is hard
Differing opinions on how severe the issue is (cost vs. benefit):
http://stew.vireo.org/posts/I-hate-akka/
https://www.reddit.com/r/scala/comments/2ruskl/i_hate_akka/

Solution coming: Akka Typed
• Unreleased and experimental
• Defined protocols with types for:
• ActorSystem!
• No user guardian, define your own!
• Actor
• ActorRef
• .tell (!)
• .ask (?)
• More information:
• http://doc.akka.io/docs/akka/snapshot/scala/typed.html
• http://blog.scalac.io/2015/04/30/leszek-akka-typed.html

In the meantime: Don’t use Actors for everything
• IMHO: Benefits outweigh the cost of the loss of type safety in
certain applications
• What do we use instead?
• Scala Futures are extremely powerful
• Futures compose nicer than Actors
• What about supervision and „let it crash”?
• Future’s handle failure also
24

Problem #2: Hellish code complexity
25

var hell
• Actors have too much mutable state
• Our worst scenario: Actor with 300 lines and 20 vars
• Hard to reason about state when everything is “global” inside
the Actor
• How do you initialize state that is only used in some
behaviours?
• var something: SomeType = _
• NPE
• var something: Option[SomeType] = None
• Always need to “getOrElse”
26

become/unbecome hell
• Pushing and popping state on a stack
• context.become(behavior: Receive, discardOld: Boolean = true)
• context.unbecome()
• “context.become” isn’t enforced to be called last
• You use methods to keep things short, but there will be multiple
methods trying to modify the behaviour
• i.e. you could end up inadvertently overwriting behavior
• One place: context.become(handleCoolMessages orElse waitForNewRequests)
• Somewhere else: context.become(waitForNewRequests, discardOld = true)
• When things blow up you have no idea how you got there
27

class Receptionist extends Actor {
def receive = waiting
def waiting: Receive = {
case Api.Scrape(url, depth) =>
context.become(next(Vector(Job(sender, url, depth))))
}
def running(queue: Vector[Job]): Receive = LoggingReceive {
case Result(links) =>
// …
context.become(next(queue.tail))
context.become(enqueue(queue, Job(sender, url, depth)))
case Terminated(_) =>
// …
}
def enqueue(queue: Vector[Job], job: Job): Receive = LoggingReceive {
// …
running(queue)
}
def next(queue: Vector[Job]): Receive = LoggingReceive {
// …
running(queue)
}
}
28

class Receptionist extends Actor {
def receive = waiting
def waiting: Receive = {
context.become(next(Vector(Job(sender, url, depth))))
}
def running(queue: Vector[Job]): Receive = LoggingReceive {
case Result(links) =>
// …
context.become(enqueue(queue, Job(sender, url, depth)))
case Terminated(_) =>
// …
}
def enqueue(queue: Vector[Job], job: Job): Receive = LoggingReceive {
// …
running(queue)
}
def next(queue: Vector[Job]): Receive = LoggingReceive {
// …
running(queue)
}
}
29
Behavior changes
all over the place

Debug hell
• Debugging Actors is hard
• Stacktraces lack meaning
• Need to have a lot of boilerplate utility code:
• What was the message?
• Who sent the message?
• What’s my internal state?
31

[ERROR] [akka://simple-actor/user/receptionist/controller-1/$a] Exception happened
java.lang.Exception: exception happened here
at meetup.akka.simple.Getter$$anonfun$receive$1$$anonfun$1.apply(Getter.scala:37)
at meetup.akka.simple.Getter$$anonfun$receive$1$$anonfun$1.apply(Getter.scala:37)
at scala.util.Try$.apply(Try.scala:191)
at meetup.akka.simple.Getter$$anonfun$receive$1.applyOrElse(Getter.scala:37)
at akka.actor.Actor$class.aroundReceive(Actor.scala:465)
at meetup.akka.simple.Getter.aroundReceive(Getter.scala:16)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)
at akka.actor.ActorCell.invoke(ActorCell.scala:487)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238)
at akka.dispatch.Mailbox.run(Mailbox.scala:220)
at akka.dispatch.ForkJoinExecutorConfigurator
$AkkaForkJoinTask.exec(AbstractDispatcher.scala:393)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:
107) 32

We need Logging for Debugging
• Akka provides:
• with ActorLogging
• log.info(…)
• LoggingRecieve: def receive = LoggingReceive { … }
• Lifecycle logging: akka.actor.debug.lifecycle=true
• Autorecieve logging: akka.actor.debug.autoreceive=true
• Issue:
• Akka only provides logging of actual messages per Receive block (has
pros and cons)
• If you missed one => good luck debugging issues around it in
production
33

[DEBUG] [akka://simple-actor/user/receptionist] started(meetup.akka.simple.Receptionist@14ba772)
[DEBUG] [akka://simple-actor/user/receptionist/controller-1] started
(meetup.akka.simple.Controller@38a5d7)
[DEBUG] [akka://simple-actor/user/receptionist] now supervising Actor[akka://simple-
actor/user/receptionist/controller-1#4232237]
[DEBUG] [akka://simple-actor/user/receptionist/controller-1] now watched by
Actor[akka://simple-actor/user/receptionist#1565954732]
[DEBUG] [akka://simple-actor/user/receptionist/controller-1] received handled
message Check(http://doc.akka.io/docs/akka/2.3.5/intro/what-is-akka.html,1)
Logging example
34

“Everything* is a Finite State Machine.”**
-me after 7 beers
* not everything 
** please do not quote me
38

Code: https://github.com/krivachy/AkkaWithFsm
39

FSM concepts
object Receptionist {
object Internal {

sealed trait State
case object Sleeping extends State
case object Processing extends State

sealed trait Data
case class NoQueue(requestId: Int = 0) extends Data
case class Queue(currentRequestId: Int, items: Vector[Job]) extends Data

}
}
40
States explicitly
defined – just typed
singletons
Data used in states
explicitly defined

class Receptionist 
extends FSM[Internal.State, Internal.Data] {…}

Define handlers for all states
// Initialize with data
startWith(Sleeping, NoQueue()) 
// Handlers for states
when(Sleeping)(enqueueNewRequest)
when(Processing) ( 
processResult 
orElse enqueueNewRequest
orElse reportError
)
42
Isn’t this
nice?

Behaviors
43
• Behavior
• StateFunction: PartialFunction[Event, State]
• Event
• Event(incomingMessage, stateData) => …
• State transitions:
• goto/stay (nextState) using (data) forMax(timeout) replying
(message)

def enqueueNewRequest: StateFunction = {
case Event(Api.Scrape(url, depth), NoQueue(requestId)) =>
// …
case Event(Api.Scrape(url, depth), queue: Queue) =>
if (queue.items.size > 3) {
stay replying Api.Failed(url)
} else {
goto(Processing) using
Queue(queue.currentRequestId, queue.items :+ Job(sender(), url,
depth))
}
}
Message Data in state
State changes
Data changes

Monitoring state transitions
onTransition {
case Idle -> Active => setTimer("timeout", Tick, 1 second, true)
case Active -> _ => cancelTimer("timeout")
case x -> Idle => log.info("entering Idle from " + x)
}
monitoredActor ! SubscribeTransitionCallBack(self)
def recieve = {
case Transition(monitoredActor, oldState, newState) =>
if (newState == Errored) alert.raiseAlert(...)
}
override def postStop() = {
monitoredActor ! UnsubscribeTransitionCallBack(self)
}
Internal:
External:
45

Handling failure
whenUnhandled {
case Event(any, data) =>
val logUpToHere = prettyPrint(getLog)
log.error(s"Unhandled event: ${any}n${logUpToHere}")
stay()
}
46

Handling failure
def handleFailureMessage: StateFunction = {
case Event(Status.Failure(cause), _) =>
log.error(s"Failed to GET $url", cause)
stop(FSM.Failure(cause))
}
onTermination {
case StopEvent(FSM.Normal, state, data) => ???
case StopEvent(FSM.Shutdown, state, data) => ???
case StopEvent(FSM.Failure(cause), state, data) => ???
}

Result is simplicity
class Receptionist extends FSM[Internal.State, Internal.Data] {
startWith(Sleeping, NoQueue())
when(Sleeping)(enqueueNewRequest)
when(Processing) (processResult orElse enqueueNewRequest orElse reportError)
def enqueueNewRequest: StateFunction = ???
def processResult: StateFunction = ???
def reportError: StateFunction = ???
whenUnhandled { ??? }
initialize()
}
48
Possible states
of Actor explicit
Behavior self-
contained
Error handling
common

Hell status
var ✔
No more mutable global state
inside Actors. Everything is
typed to the specific State.
become/unbecome ✔
All methods have to end in a
state transition. States are
clearly defined what they do.
debug/logging ?
49

LoggingFSM
• Remembers state transitions:
• Can override size of history (logDepth defaults to 8)
• Works super well with: onTermination
• Debug logging: akka.actor.debug.fsm=true
• Automatically logs important Events: message + internal
data
• Logs state transitions
• Use with: akka.actor.debug.lifecycle=true
51

Debug log example
[DEBUG] [akka://fsm/user/receptionist] processing
Event(Result(Set([…])),Queue(1,Vector(Job(Actor[akka://fsm/
system/testActor1#758674372],http://doc.akka.io/docs/akka/
2.3.5/intro/what-is-akka.html,1)))) from Actor[akka://fsm/
user/receptionist/controller-1#4232237]
[DEBUG] [akka://fsm/user/receptionist/controller-1] transition
CollectingResults -> Completed
[DEBUG] [akka://fsm/user/receptionist] transition Processing -
> Sleeping
[DEBUG] [akka://fsm/user/receptionist/controller-1] stopped
52
Internal state
logged
Incoming
message logged

[ERROR] [akka://fsm/user/receptionist] Unhandled event: some string
Last 8 entries leading up to this point:
in state: Sleeping
with data: NoQueue(2)
received: Scrape(http://non-existent.link,5)
in state: Processing
with data: Queue(3,Vector(Job(Actor[akka://fsm/system/testActor1#758674372],http://
non-existent.link,5)))
received: Result(Set(http://non-existent.link))
[…]
in state: Processing
with data: Queue(4,Vector(Job(Actor[akka://fsm/system/testActor1#758674372],http://
non.existent1,0), Job(Actor[akka://fsm/system/testActor1#758674372],http://
non.existent4,0)))
received: some string
53
Awesome
history of what
happened

Hell status
var ✔
No more mutable global state
inside Actors. Everything is
typed to the specific State.
become/unbecome ✔
All methods have to end in a
state transition. States are
clearly defined what they do.
debug/logging ✔
FSM does all the debug
logging we would ever need.
54

…but you replace one set of
problems for another!

Akka Typed 
will help solve some issues

My experiences
•Needs a new mindset
•Unexperienced developers + learning curve +
easy to make mistakes = headaches
•Use Actors only when really needed
• Too much testing of protocols and communication
•Try to make all Actors FSM Actors
•Helps solve issues around Actor complexity
@akoskrivachy https://github.com/krivachy/AkkaWithFsm

The dark side of Akka and the remedy

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to The dark side of Akka and the remedy

Similar to The dark side of Akka and the remedy (20)

Recently uploaded

Recently uploaded (20)

The dark side of Akka and the remedy