In order to be successful with asynchronous programming, when coming from synchronous execution models you need to change your mindset and look at things from a slightly different perspective. In order to use Akka at it's best, you will have to change the way you think about application design (loosen coupling in space and time between components), and re-think what you've maybe learned in the past.
In this talk we uncover a number of rules that serve as a guide in designing concurrent distributed applications, how those apply to Akka, and how they can help you in daily app development.
Aimed at developers through architects, Akka team happy hAkker, Konrad Malawski, bends your parameters with regards to application design and asynchronous execution models.
4. Why such talk?
1 : One actor is no Actor
2 : Structure your Actors
3 : Name your Actors
4 : âMatrix of mutability (Pain)â
5 : Blocking needs careful management
6 : Never Await, for/ïŹatMap instead!
7 : Avoid Java Serialization
7.5 : Trust no-one, benchmark everything!
Agenda
8 : Let it Crash!
9 : Backoff Supervision
10 : Design using State Machines
11 : Cluster Convergence and Joining
12 : Cluster Partitions and âDownâ
13 : Akka is a Toolkit.
14 : Happy Hakking, Community!
Questions?
5.
6. âThe Tao / Zen of Programmingâ
Talk title loosely based on
the âTao of Programmingâ book
by Goeffrey James (1987).
7. âThe Tao / Zen of Programmingâ
And the follow-up book
âZen of Programmingâ.
8. âThe Tao / Zen of Programmingâ
Available here: http://www.mit.edu/~xela/tao.html
Series of nine âbooksâ,
stories about an apprentice programmer and his sensei.
Thus spake the Master Programmer:
âWithout the wind, the grass does not move.
Without software hardware is useless.â
11. The Zen of Akka
Is best explained as a way of thinking about Architecture.
Akka provides building blocks, with speciïŹc semantics.
12. The Zen of Akka
Is best explained as a way of thinking about Architecture.
Akka provides building blocks, with speciïŹc semantics.
Actors are cheap â so they can be 1:1 for a user, or wallet etc
Actors are referentially transparent â can scale-out trivially
Actors encapsulate state â avoiding global state
Actors are engines â
Streams / Streaming HTTP / Cluster Sharding / Distributed DataâŠ
â all using are Actors as engines, high-level Architectural help.
15. 1 : One actor is no Actor
If you have only one actor then it can onlyâŠ
1. Reply
2. Drop the message (âignoreâ
3. Schedule another message to self
So weâre not really making any use of its
parallelism or concurrency capabilities.
21. 1 : One actor is no Actor
- Actors are meant to work together.
- An Actor should do one thing and do it very well
- then talk to other Actors to do other things for it.âš
- Child Actors usually used for workers or âtasksâ etc.âš
- Avoid using `actorSelection`,âš
introduce Actors to each other.
27. 3 : Name your Actors
// default
context.actorOf(childProps) // "$a", "$b", "$c"
Default names are: BASE64(sequence_nr++)
Hereâs why:
- cheap to generate
- guarantees uniqueness
- less chars than plain numbers
28. 3 : Name your Actors
// default: naming is BASE64(sequential numbers)
context.actorOf(childProps) // "$a", "$b", "$c"
// better: but not very informative...
context.actorOf(childProps, nextFetchWorkerName) // "fetch-worker-1", "fetch-worker-2"
private var _fetchWorkers: Int = 0
private def nextFetchWorkerName: String = {
_fetchWorkers += 1
sâfetch-worker-${_fetchWorkers}â
}
Sequential names are a bit better sometimes.
29. 3 : Name your Actors
// default: naming is BASE64(sequential numbers)
context.actorOf(childProps) // "$a", "$b", "$c"
// better: but not much informative...
context.actorOf(childProps, nextFetchWorkerName) // "fetch-worker-1", "fetch-worker-2"
private var _fetchWorkers: Int = 0
private def nextFetchWorkerName: String = {
_fetchWorkers += 1
sâfetch-worker-${_fetchWorkers}â
}
abstract class SeqActorName {
def next(): String
def copy(name: String): SeqActorName
}
object SeqActorName {
def apply(prefix: String) = new SeqActorNameImpl(prefix, new AtomicLong(0))
}
final class SeqActorNameImpl(val prefix: String, counter: AtomicLong)
extends SeqActorName {
def next(): String = prefix + '-' + counter.getAndIncrement()
def copy(newPrefix: String): SeqActorName = new SeqActorNameImpl(newPrefix, counter)
}
If you use this pattern a lot, hereâs a simple encapsulation of it:
30. 3 : Name your Actors
// default: naming is BASE64(sequential numbers)
context.actorOf(childProps) // "$a", "$b", "$c"
// better: but not much informative...
context.actorOf(childProps, nextFetchWorkerName) // "fetch-worker-1", "fetch-worker-2"
private var fetchWorkers: Int = 0
private def nextFetchWorkerName: String = {
fetchWorkers += 1
s"fetch-worker-$fetchWorkers"
}
// BEST: proper names, based on useful information
context.actorOf(childProps, fetcherName(videoUrl)) // "fetch-yt-MRCWy2E_Ts", ...
def fetcherName(link: Link) = link match {
case YoutubeLink(id, metadata) => s"fetch-yt-$id"
case DailyMotionLink(id, metadata) => s"fetch-dm-$id"
case VimeoLink(id, metadata) => s"fetch-vim-$id"
}
Meaningful names are the best!
31. 3 : Name your Actors
Meaningful names are the best!
import akka.actor.OneForOneStrategy
import akka.actor.SupervisorStrategy._
import scala.concurrent.duration._
// ... extends Actor with ActorLogging {
override def supervisorStrategy: SupervisorStrategy =
OneForOneStrategy(maxNrOfRetries = 10, withinTimeRange = 1.minute) {
case ex: Exception
log.warning("Child {} failed with {}, attempting restart...",
sender().path.name,
ex.getMessage)
Restart
}
The name of the failed child Actor!
32. 3 : Name your Actors
Meaningful names are the best!
import akka.actor.OneForOneStrategy
import akka.actor.SupervisorStrategy._
import scala.concurrent.duration._
// ... extends Actor with ActorLogging {
override def supervisorStrategy: SupervisorStrategy =
OneForOneStrategy(maxNrOfRetries = 10, withinTimeRange = 1.minute) {
case ex: Exception
log.warning("Child {} failed with {}, attempting restart...",
sender().path.name,
ex.getMessage)
Restart
}
The name of the failed child Actor!
// BAD ââ String ALWAYS built
log.debug(s"Something heavy $generateId from $physicalAddress")
// GOOD! ââ String built only when DEBUG level is ON
log.debug("Something heavy {} from {}", generateId, physicalAddress)
Side note: always use {} log formatting (or macros), not sââ
41. 5 : Blocking needs careful management
Blocking operations are really bad.
Actors are all about resource sharing, and if someone is âbehaving
badlyâ it hurts everyone.
Here is an example how blocking can grind an app to a halt.
Next weâll see how to avoid that⊠even if we have to live with the
blocking code.
42. 5 : Blocking needs careful management
In simple terms:
Blocking is bad because instead of doing something else,
we just wait and do nothing (wasting CPU time)âŠ
44. 5 : Blocking needs careful management
Having that said, itâs not a bad question. Letâs investigate.
45. 5 : Blocking needs careful management
// BAD! (due to the blocking in Future):
implicit val defaultDispatcher = system.dispatcher
val routes: Route = post {
complete {
Future { // uses defaultDispatcher
Thread.sleep(5000) // will block on the default dispatcher,
System.currentTimeMillis().toString // starving the routing infra
}
}
}
46. 5 : Blocking needs careful management
// BAD! (due to the blocking in Future):
implicit val defaultDispatcher = system.dispatcher
val routes: Route = post {
complete {
Future { // uses defaultDispatcher
Thread.sleep(5000) // will block on the default dispatcher,
System.currentTimeMillis().toString // starving the routing infra
}
}
}
47. 5 : Blocking needs careful management
// application.conf
my-blocking-dispatcher {
type = Dispatcher
executor = âthread-pool-executor"
thread-pool-executor {
// in Akka previous to 2.4.2:
core-pool-size-min = 16
core-pool-size-max = 16
max-pool-size-min = 16
max-pool-size-max = 16
// or in Akka 2.4.2+
ïŹxed-pool-size = 16
}
throughput = 100
}
48. 5 : Blocking needs careful management
// GOOD (due to the blocking on a dedicated dispatcher):
implicit val blockingDispatcher = system.dispatchers.lookup("my-blocking-dispatcher")
val routes: Route = post {
complete {
Future { // uses the good "blocking dispatcher" that we configured,
// instead of the default dispatcher â the blocking is isolated.
Thread.sleep(5000)
System.currentTimeMillis().toString
}
}
}
49. 5 : Blocking needs careful management
The âNever block!â mantra sounds cool,
but actually what we mean by it is âblocking needs careful managementâ.
We use the âbulkheadâ pattern separate out potentially blocking
behaviours to their independent dispatchers (and should always do so).
http://stackoverïŹow.com/questions/34641861/akka-http-blocking-in-a-future-blocks-the-server/34645097#34645097
51. 6 : Never Await, for/flatMap instead!
// ... extends Actor {
import context.dispatcher
import scala.concurrent.duration._
import scala.concurrent.Await // bad sign!
// BAD!!!
val fThings: Future[Things] = computeThings()
val t: Things = Await.result(fThings, atMost = 3.seconds)
val d: Details = Await.result(moreDetailsFor(t), atMost = 3.seconds)
52. 6 : Never Await, for/flatMap instead!
// ... extends Actor {
import context.dispatcher
import scala.concurrent.duration._
import scala.concurrent.Await // bad sign!
// BAD!!!
val fThings: Future[Things] = computeThings()
val t: Things = Await.result(fThings, atMost = 3.seconds)
val d: Details = Await.result(moreDetailsFor(t), atMost = 3.seconds)
// Good:
val fThingsWithDetails = for {
t <- computeThings()
d <- moreDetailsFor(t)
} yield t -> d
fThingsWithDetails foreach {
case (things, details) => // case (things: Things, details: Details) =>
println(s"$things with $details")
}
53. 6 : Never Await, for/flatMap instead!
// ... extends Actor {
import context.dispatcher
import scala.concurrent.duration._
import scala.concurrent.Await // bad sign!
// BAD!!!
val fThings: Future[Things] = computeThings()
val t: Things = Await.result(fThings, atMost = 3.seconds)
val d: Details = Await.result(moreDetailsFor(t), atMost = 3.seconds)
// Good:
val fThingsWithDetails = for {
t <- computeThings()
d <- moreDetailsFor(t)
} yield t -> d
fThingsWithDetails foreach {
case (things, details) => // case (things: Things, details: Details) =>
println(s"$things with $details")
}
// adding timeout:
val timeoutFuture = akka.pattern.after(3.seconds, context.system.scheduler) {
Future.failed(new TimeoutException("My timeout details..."))
}
Future.firstCompletedOf(fThingsWithDetails :: timeoutFuture :: Nil) foreach {
case (things, details) => // case (things: Things, details: Details) =>
println(s"$things with $details")
}
55. 7 : Avoid Java Serialization
Java Serialization is the default one in Akka, since itâs easy to
get started with it â no conïŹguration needed.
If you need performance and are running on multiple nodes,
you must change the serialization.
Popular formats are ProtoBuf or Kryo.
Kryo is easier, but harder to evolve schema with.
ProtoBuf is harder to maintain but great schema evolution.
56. 7 : Avoid Java Serialization
Benchmarking serialization impact on âping pongâ case.
(Two actors sending a message between them.)
in-process messaging, super fast.âš
no serialization overhead.
57. 7 : Avoid Java Serialization
more work => increased latency => decreased throughput.
over-the-network messaging,
slower due to network and serialization.
58. 7 : Avoid Java Serialization
Java Serialization is known to be:
very slow & footprint heavy
59. 7 : Avoid Java Serialization
It is on by default in Akka⊠Why?
a) zero setup => simple to âplay aroundâ
b) historical reasons - hard to remove the default
Since 2.4 a warning is logged:âš
WARNING: Using the default Java serializer for class [{}] which is not recommended
because of performance implications. Use another serializer or disable this warning
using the setting 'akka.actor.warn-about-java-serializer-usage'
60. 7 : Avoid Java Serialization
sbt> jmh:run âš
-f 1
-tu us âš
-wi 20
-i 10
-jvm /home/ktoso/opt/jdk1.8.0_65/bin/java
-jvmArgsAppend -XX:+PreserveFramePointer
-bm avgt
.*pingPong.*
[info] # JMH 1.10.3 (released 184 days ago, please consider updating!)
[info] # VM version: JDK 1.8.0_65, VM 25.65-b01
[info] # VM invoker: /home/ktoso/opt/jdk1.8.0_65/bin/java
[info] # VM options: -XX:+PreserveFramePointer
[info] # Warmup: 20 iterations, 5 s each
[info] # Measurement: 10 iterations, 1 s each
[info] # Timeout: 10 min per iteration
[info] # Threads: 1 thread, will synchronize iterations
[info] # Benchmark mode: Average time, time/op
[info] # Benchmark: akka.actor.ForkJoinActorBenchmark.pingPong
[info] # Parameters: (serializer = java)
github.com/ktoso/sbt-jmh
openjdk.java.net/projects/code-tools/jmh/
63. 7 : Avoid Java Serialization
----sr--model.Order----h#-----J--idL--customert--Lmodel/Customer;L--descriptiont--Ljava/lang/String;L--orderLinest--Ljava/util/List;L--totalCostt--Ljava/
math/BigDecimal;xp--------ppsr--java.util.ArrayListx-----a----I--sizexp----w-----sr--model.OrderLine--&-1-S----I--lineNumberL--costq-~--L--descriptionq-
~--L--ordert--Lmodel/Order;xp----sr--java.math.BigDecimalT--W--(O---I--scaleL--intValt--Ljava/math/BigInteger;xr--java.lang.Number-----------xp----sr--
java.math.BigInteger-----;-----I--bitCountI--bitLengthI--ïŹrstNonzeroByteNumI--lowestSetBitI--signum[--magnitudet--[Bxq-~----------------------ur--[B------
T----xp----xxpq-~--xq-~--
Java Serialization
final case class Order(id: Long, description: String, totalCost: BigDecimal,
orderLines: ArrayList[OrderLines], customer: Customer)
<order id="0" totalCost="0"><orderLines lineNumber="1" cost="0"><order>0</order></orderLines></order>XMLâŠ!
{"order":{"id":0,"totalCost":0,"orderLines":[{"lineNumber":1,"cost":0,"order":0}]}}JSONâŠ!
------java-util-ArrayLis-----model-OrderLin----java-math-BigDecima---------model-Orde-----KryoâŠ!
Excellent post by James Sutherland @
http://java-persistence-performance.blogspot.com/2013/08/optimizing-java-serialization-java-vs.html
64. 7 : Avoid Java Serialization for Persistence!!!
Java Serialization is a horrible idea if youâre going to store the
messages for a long time.
For example, with Akka Persistence we store events âforeverâ.
Use a serialization format that can evolve over time in a
compatible way. It can be JSON or ProtocolBuffers (or Thrift
etc).
65. 7.5 : Trust no-one, benchmark everything!
Always measure and benchmark properly
before judging performance of a tool / library.
Benchmarking is often very hard,
use the right tools:
- JMH (for Scala via: ktoso/sbt-jmh)
-YourKit / JProïŹler / âŠ
- Linux perf_events
7.5
66. 8 : Let it Crash! Supervision, Failures & Errors
67. http://www.reactivemanifesto.org/
8 : Let it Crash! Supervision, Failures & Errors
Error
⊠which is an expected and coded-for conditionâfor
example an error discovered during input validation, that
will be communicated to the client âŠ
Failure
⊠is an unexpected event within a service that
prevents it from continuing to function normally. âš
A failure will generally prevent responses to the current,
and possibly all following, client requests.
79. 9 : Backoff Supervision
IF we allowed immediate restartsâŠ
we could end up in âinïŹnite replay+fail hellâ.
(we donât. since Persistence went stable in 2.4.x)
86. 10 : Design using State Machines
def receive = {
case Thingy() =>
// ...
case AnotherThingy() =>
// ...
case DoOtherThings() =>
// ...
case PleaseGoAway() =>
// ...
case CarryOn() =>
// ...
case MakeSomething() =>
// ...
// ...
}
87. 10 : Design using State Machines
def receive = {
case Thingy() =>
// ...
case AnotherThingy() =>
// ...
case DoOtherThings() =>
// ...
case PleaseGoAway() =>
// ...
case CarryOn() =>
// ...
case MakeSomething() =>
// ...
// ...
}
Good:
Actors avoid the âpyramid of doomâ.
Pyramid of doom in some
async programming styles.
88. 10 : Design using State Machines
def receive = {
case Thingy() =>
// ...
case AnotherThingy() =>
// ...
case DoOtherThings() =>
// ...
case PleaseGoAway() =>
// ...
case CarryOn() =>
// ...
case MakeSomething() =>
// ...
// ...
}
That well works because
âeverything is a messageâ:
89. 10 : Design using State Machines
def receive = awaitingInstructions
def awaitingInstructions: Receive =
terminationHandling orElse {
case CarryOn() =>
// ...
case MakeSomething(metadata) =>
// ...
context become makeThings(meta)
}
def makeThings(metadata: Metadata): Receive =
terminationHandling orElse {
case Thingy() =>
// make a thingy ...
case AnotherThingy() =>
// make another thingy ...
case DoOtherThings(meta) =>
// ...
context become awaitingInstructions
}
def terminationHandling: Receive = {
case PleaseGoAway() =>
// ...
context stop self
}
DoOtherThings
MakeSomething
90. 10 : Design using State Machines
We also provide an FSM (Finite State Machine) helper trait.
You may enjoy it sometimes, give it a look.
DoOtherThings
MakeSomething
http://doc.akka.io/docs/akka/2.4.1/scala/fsm.html
class Buncher extends FSM[State, Data] {
startWith(Idle, Uninitialized)
when(Idle) {
case Event(SetTarget(ref), Uninitialized) =>
stay using Todo(ref, Vector.empty)
}
// transition elided ...
when(Active, stateTimeout = 1 second) {
case Event(Flush | StateTimeout, t: Todo) =>
goto(Idle) using t.copy(queue = Vector.empty)
}
// unhandled elided ...
initialize()
}
92. 11 : Cluster Convergence and Joining
http://doc.akka.io/docs/akka/2.4.1/common/cluster.html
Cluster Gossip Convergence
When a node can prove that the cluster state it is observing
has been observed by all other nodes in the cluster.
93. 11 : Cluster Convergence and Joining
http://doc.akka.io/docs/akka/2.4.1/common/cluster.html
Cluster Gossip Convergence
When a node can prove that the cluster state it is observing
has been observed by all other nodes in the cluster.
Convergence is required for âLeader actionsâ,
which include Join-ing and Remove-ing a node.
Down-ing can happen without convergence.
94. 11 : Cluster Convergence and Joining
http://doc.akka.io/docs/akka/2.4.1/common/cluster.html
95. 11 : Cluster Convergence and Joining
http://doc.akka.io/docs/akka/2.4.1/common/cluster.html
akka.cluster.allow-weakly-up-members=on
109. 12 : Cluster Partitions and âDownâ
Iâm going
homeâŠ
110. 12 : Cluster Partitions and âDownâ
Make sure the others have heard you say goodbye before you leave.
Vanishes immediately.
111. 12 : Cluster Partitions and âDownâ
Make sure the others have heard you say goodbye before you leave.
âWhereâs Bill?
I did not hear him say Goodbye!â
âFailure Detectorâ used to determine UNREACHABLE.
112. 12 : Cluster Partitions and âDownâ
Failure detection only triggers âUNREACHABLEâ.
Nodes can come back from that state.
113. 12 : Cluster Partitions and âDownâ
Declaring DOWN is done by either timeouts (which is rather unsafe) [auto-downing].
Or by âvotingâ or âmajorityâ among the members of the cluster [split-brain-resolver].
116. 12 : Cluster Partitions and âDownâ
You
are
already
dead.
117. 12 : Cluster Partitions and âDownâWhy do we do that?
In order to guarantee consistency via
âsingle writer principleâ.
Note:
Akka Distributed Data has no need for âsingle
writerâ, itâs CRDT based. But itâs harder to model
things as CRDT, so itâs a trade off.
You
are
already
dead.
118. 12 : Cluster Partitions and âDownâ
Notice that we do not mention âQuarantinedâ.
That is a state in Akka Remoting, not Cluster.
Itâs a terminal state from which one can never recover.
TL;DR;
use Akka Cluster instead of Remoting.
itâs pretty much always the thing you need (better than remoting).
119. 13 : A fishing rod is a Tool. Akka is a Toolkit.
120. 13 : A fishing rod is a Tool. Akka is a Toolkit.
Akka strives is Toolkit,
not a Framework.
âGive a man a ïŹsh and you feed him for a day
teach a man to ïŹsh and you feed him for a lifetime.â
121. 13 : A fishing rod is a Tool. Akka is a Toolkit.
Akka strives is Toolkit,
not a Framework.
Play is a Framework,
Lagom is a Framework.
122. 13 : Akka is a Toolkit, pick the right tools for the job.
âConstraints Liberate,
Liberties Constrainâ
Runar Bjarnason
Runarâs excellent talk @ Scala.World 2015
123. 13 : Akka is a Toolkit, pick the right tools for the job.
Runarâs excellent talk @ Scala.World 2015
The less powerful abstraction
must be built on top of
more powerful abstractions.
124. 13 : Akka is a Toolkit, pick the right tools for the job.
Runarâs excellent talk @ Scala.World 2015
Asynchronous processing toolbox:
125. 13 : Akka is a Toolkit, pick the right tools for the job.
Runarâs excellent talk @ Scala.World 2015
Asynchronous processing toolbox:
126. 13 : Akka is a Toolkit, pick the right tools for the job.
Asynchronous processing toolbox:
127. 13 : Akka is a Toolkit, pick the right tools for the job.
Single value, no streaming by deïŹnition.
Local abstraction.âš
Execution contexts.
128. 13 : Akka is a Toolkit, pick the right tools for the job.
Mostly static processing layouts.
Well typed and Back-pressured!
129. 13 : Akka is a Toolkit, pick the right tools for the job.
Plain Actorâs younger brother, experimental.
Location transparent, well typed.
Technically unconstrained in actions performed
130. 13 : Akka is a Toolkit, pick the right tools for the job.
Runarâs excellent talk @ Scala.World 2015
Location transparent.
Various resilience mechanisms.
(watching, persistent recovering, migration, pools)
Untyped and unconstrained in actions performed.
142. 14 : Happy hAkking, Community!
akka.io â website
github.com/akka/akka/issues â help out!
âcommunity-contribâ or âsmallâ for starters
groups.google.com/group/akka-user â mailing list
gitter.im/akka/akka â chat about using Akka
gitter.im/akka/dev â chat about developing Akka
144. Thanks!
ktoso @ typesafe.com
twitter: ktosopl
github: ktoso
team blog: letitcrash.com
home: akka.io
Thus spake the Master Programmer:
âAfter three days without programming,
life becomes meaningless.â