In this special guest webinar with Akka expert and Reactive System Consultant, Manuel Bernhardt, we review Akka 2.6 release highlights and a selection of 6 former anti-patterns that have now been rendered impossible by design.
4. WHY?
> collected recuring "worst practices"
over the years 1
> ranges from beginner mistakes to
design issues that can affect a
project over years
> most of it has been adressed in Akka
2.6! Let's have a look at 6 of them
1
https://manuel.bernhardt.io/akka-anti-patterns
Webinar - https://manuel.bernhardt.io - @elmanu
5. MANUEL.BERNHARDT.IO
> Helping companies to get started with
reactive systems...
> ... or to solve critical production
issues
> Lightbend consulting and training
partner
Webinar - https://manuel.bernhardt.io - @elmanu
6. MANUEL.BERNHARDT.IO
> Helping companies to get started with
reactive systems...
> ... or to solve critical production
issues
> Lightbend consulting and training
partner
Webinar - https://manuel.bernhardt.io - @elmanu
7. AKKA 2.6
> previous major release (2.5.0) in April 2017
> introduces Akka Typed API, but does not deprecate the
Classic one
> Artery as default transport
> many, many improvements and fixes
Webinar - https://manuel.bernhardt.io - @elmanu
11. > quite often, actor systems are built without giving
much thought to the message protocol itself
> this works quite well in the beginning, but can become
quite a burden in the long run
Webinar - https://manuel.bernhardt.io - @elmanu
12. HOW THINGS MAY GO WRONG
> tight coupling: message re-use that makes it harder to
evolve over time
> responsibilities: responsibilities that belong together
may be split accross actors over time
> inefficiencies: "organic" growth of an actor system
makes it easy to loose track of how information is
exchanged, can lead to unnecessary message exchanges
Webinar - https://manuel.bernhardt.io - @elmanu
13. It is really difficult to evolve protocols in the classic
API due its dynamic nature / very open definition
Webinar - https://manuel.bernhardt.io - @elmanu
14. "NOT DEFINING A PROTOCOL" IN AKKA 2.6 /
TYPED
⠀
Webinar - https://manuel.bernhardt.io - @elmanu
15. "NOT DEFINING A PROTOCOL" IN AKKA 2.6 /
TYPED
> you can't.
Webinar - https://manuel.bernhardt.io - @elmanu
16. "NOT DEFINING A PROTOCOL" IN AKKA 2.6 /
TYPED
> you can't.
> Akka Typed forces you to define a protocol
> you need to specify:
> what type of message an actor can handle
> what type of response a message expects
Webinar - https://manuel.bernhardt.io - @elmanu
17. TYPE OF MESSAGE AN ACTOR CAN HANDLE
// marker of the message protocol
sealed trait ConfigurationCommand
// actor supporting this protocol
class Configuration(
context: ActorContext[ConfigurationCommand]
) extends AbstractBehavior[ConfigurationCommand] { ... }
Webinar - https://manuel.bernhardt.io - @elmanu
18. TYPE OF MESSAGE AN ACTOR EXPECTS (1)
// a message of the Configuration protocol
final case class RetrieveConfiguration(
merchantId: MerchantId,
replyTo: ActorRef[ConfigurationResponse]
) extends ConfigurationCommand
Webinar - https://manuel.bernhardt.io - @elmanu
19. TYPE OF MESSAGE AN ACTOR EXPECTS (2)
sealed trait ConfigurationResponse
final case class ConfigurationFound(
configuration: MerchantConfiguration
) extends ConfigurationResponse
Webinar - https://manuel.bernhardt.io - @elmanu
20. HOW DOES THIS CHANGE THE GAME?
> the beginner mistake of sending messages to the wrong
actor can no longer happen
> message paths are now checked by the compiler
> you can infer the protocol flow from the
ActorRef[Message] included in the messages
Webinar - https://manuel.bernhardt.io - @elmanu
21. sealed trait BankingCommand
final case class Login(
credentials: Credentials,
replyTo: ActorRef[LoginResponse]
) extends BankingCommand
sealed trait LoginResponse
final case class Authenticated(
session: ActorRef[SessionCommand]
) extends LoginResponse
sealed trait SessionCommand
final case class CreateTransaction(
transaction: Transaction
) extends SessionCommand
Webinar - https://manuel.bernhardt.io - @elmanu
25. > in the actor model, actors handle messages one after
another
> it is safe to use mutable state inside of an actor
> unless you do the following
Webinar - https://manuel.bernhardt.io - @elmanu
26. CLASSIC API
class TransactionManager(processor: ActorRef) extends Actor {
var transactions = List.empty[Transaction]
def receive: Receive = {
case ProcessTransaction(transactionRequest) =>
val response: Future[ProcessingResponse] =
processor ? Process(transactionRequest)
response.map {
// here be dragons!
transactions = response.transaction :: transactions
}
}
}
Webinar - https://manuel.bernhardt.io - @elmanu
27. WHAT'S THE PROBLEM?
> using asynchronous constructs inside of the actor
(single-threaded illusion) is error-prone
> quite easy to make mistakes of this kind (I've seen it
many, many times)
> the recommended way is to use ask in combination with
the pipe pattern, but people may forget / not know
Webinar - https://manuel.bernhardt.io - @elmanu
32. "...some men aren't looking for
anything logical, like money. They
can't be bought, bullied, reasoned,
or negotiated with. Some men just
want to watch the world burn."
— Alfred Pennyworth, Batman, The Dark Knight
Webinar - https://manuel.bernhardt.io - @elmanu
34. HOW CAN THIS HAPPEN?
> calling a synchronous API
> or calling an API that calls an API that calls an API...
that calls a synchronous API
> explicitly waiting for the completion of a Future
(Await.result)
Webinar - https://manuel.bernhardt.io - @elmanu
35.
36.
37.
38.
39.
40.
41.
42. BRINGING DOWN AKKA 2.6 BY BLOCKING
⠀
Webinar - https://manuel.bernhardt.io - @elmanu
43. BRINGING DOWN AKKA 2.6 BY BLOCKING
> you can't.
Webinar - https://manuel.bernhardt.io - @elmanu
44. BRINGING DOWN AKKA 2.6 BY BLOCKING
> you can't.
> new internal dispatcher for all of Akka's internal
actors
> you can still starve your own actors, but Akka's
internal mechanisms will continue working
Webinar - https://manuel.bernhardt.io - @elmanu
45. IF YOU REALLY NEED TO BLOCK
blocking-io-dispatcher {
type = Dispatcher
executor = "thread-pool-executor"
thread-pool-executor {
fixed-pool-size = 8
}
throughput = 1
}
Webinar - https://manuel.bernhardt.io - @elmanu
46. IF YOU REALLY NEED TO BLOCK
context.spawn(
blockingBehavior,
"blocking-actor",
DispatcherSelector.fromConfig(
"blocking-io-dispatcher"
)
)
Webinar - https://manuel.bernhardt.io - @elmanu
50. HOW COULD THIS HAPPEN?
> leaving the default on
> ignoring the nagging warning messages in the logs
> affects both serialization of messages over the wire
and for persistence
Webinar - https://manuel.bernhardt.io - @elmanu
51. WHY IS IT BAD?
> performance penalty!
> poor candidate for protocol evolution - message
evolutions result in older components not able to
process them any longer
> persistance - messages and snapshots can't be
processed any longer after changes
> poor security
Webinar - https://manuel.bernhardt.io - @elmanu
53. USING JAVA SERIALIZATION IN AKKA 2.6
> you can't.
Webinar - https://manuel.bernhardt.io - @elmanu
54. USING JAVA SERIALIZATION IN AKKA 2.6
> you can't.
> ... or at least not by default
> Jackson is the new default, with a decently performing
binary serialization format (CBOR), should be good in
most cases
Webinar - https://manuel.bernhardt.io - @elmanu
55. USING JACKSON SERIALIZATION
/** marker interface **/
public interface CborSerializable {}
class Message implements CborSerializable {
public final String id;
public Message(String id) {
this.id = id;
}
}
Webinar - https://manuel.bernhardt.io - @elmanu
56. USING JACKSON SERIALIZATION
akka.actor {
serialization-bindings {
"com.example.CborSerializable" = jackson-cbor
}
}
Webinar - https://manuel.bernhardt.io - @elmanu
57. JACKSON VS. PROTOCOL BUFFERS
> protobuf as a very popular binary serialization in
Akka projects in the past
> it turns out that Jackson is a very decent alternative
and involves much less work than protobuf - also in
terms of evolutions
> definitely worth checking it out in detail before falling
back to protobuf
Webinar - https://manuel.bernhardt.io - @elmanu
64. WHAT'S THE PROBLEM?
> Akka Remoting allows to build pair-to-pair
connections between nodes / Actor Systems
> there's a lot more things to do when it comes to
building stable applications on top of the network
> not trivial
Webinar - https://manuel.bernhardt.io - @elmanu
66. DIRECTLY USING AKKA REMOTING IN AKKA
2.6
⠀
Webinar - https://manuel.bernhardt.io - @elmanu
67. DIRECTLY USING AKKA REMOTING IN AKKA
2.6
> you can't.
Webinar - https://manuel.bernhardt.io - @elmanu
68. DIRECTLY USING AKKA REMOTING IN AKKA
2.6
> you can't.
> remote watch and deployment will only work when
using the Cluster extension
Webinar - https://manuel.bernhardt.io - @elmanu
69. AKKA CLUSTER
> membership service
> primitives for building distributed
systems: membership, singleton,
sharding, CRDTs, etc.
> check out the articles series2
about
it
2
https://manuel.bernhardt.io/articles/#akka-cluster
Webinar - https://manuel.bernhardt.io - @elmanu
71. There are two ways to be fooled.
One is to believe what isn't true;
the other is to refuse to believe
what is true.
— Soren Kierkegaard
Webinar - https://manuel.bernhardt.io - @elmanu
73. GROUP MEMBERSHIP WITH A SINGLE
GROUP IS IMPOSSIBLE WHEN THERE ARE
NODES THAT ARE SUSPECTED OF
HAVING FAILED 3
3
Chandra et al: On the Impossibility of Group Membership (1996)
Webinar - https://manuel.bernhardt.io - @elmanu
74. IT WOULD BE UNWISE TO MAKE
MEMBERSHIP-RELATED DECISIONS
WHILE THERE ARE PROCESSES
SUSPECTED OF HAVING CRASHED
Webinar - https://manuel.bernhardt.io - @elmanu
75. "NETWORK PARTITIONS
DO NOT HAPPEN"
> "we have never observed a partition
in our network"
> "we have been running this
infrastructure for X months now and
nothing ever happened"
> "our network is reliable"
Webinar - https://manuel.bernhardt.io - @elmanu
76.
77.
78.
79. NETWORK PARTITIONS
> more on https://aphyr.com/posts/288-the-network-
is-reliable
> it's not a matter of if, but when.
Webinar - https://manuel.bernhardt.io - @elmanu
80. IN AKKA CLUSTER, THE LEADER CANNOT
TAKE DECISIONS SO LONG AS THERE
ARE MEMBERS THAT ARE FLAGGED AS
UNREACHABLE BY THE FAILURE
DETECTOR
Webinar - https://manuel.bernhardt.io - @elmanu
87. IF YOU USE AUTO-DOWNING IN
PRODUCTION, YOU ARE GUARANTEED TO
GET A SPLIT-BRAIN SCENARIO AT SOME
POINT IN TIME
Webinar - https://manuel.bernhardt.io - @elmanu
89. USING AUTO-DOWNING IN AKKA 2.6
> you can't.
Webinar - https://manuel.bernhardt.io - @elmanu
90. USING AUTO-DOWNING IN AKKA 2.6
> you can't.
> it's gone.
Webinar - https://manuel.bernhardt.io - @elmanu
91. USING AUTO-DOWNING IN AKKA 2.6
> you can't.
> it's gone.
> use a proper split brain resolver / preventer
Webinar - https://manuel.bernhardt.io - @elmanu
92. SPLIT BRAIN RESOLVERS
> Akka Split Brain Resolver by Lightbend
> open-source implementations (at your own risk)
> roll your own (seriously, don't do that)
Webinar - https://manuel.bernhardt.io - @elmanu
93. ANTI-PATTERN #7BEING OUT OF TOUCH WITH THE HARDWARE
Webinar - https://manuel.bernhardt.io - @elmanu
106. EXAMPLE
> kubernetes deployment with 2 CPUs /
container
> default Akka configuration - 16
threads default EC + 16 threads for
the internal dispatcher
=> 32 threads contending for 1 real
core which itself may not be dedicated
Webinar - https://manuel.bernhardt.io - @elmanu
107. BEING OUT OF TOUCH WITH THE HARDWARE
IN AKKA 2.6
⠀
Webinar - https://manuel.bernhardt.io - @elmanu
108. BEING OUT OF TOUCH WITH THE HARDWARE
IN AKKA 2.6
> you absolutely can.
Webinar - https://manuel.bernhardt.io - @elmanu
109. BEING OUT OF TOUCH WITH THE HARDWARE
IN AKKA 2.6
> you absolutely can.
> Akka can do a lot, but it can't do magic
> learn the basics, know your infrastructure
Webinar - https://manuel.bernhardt.io - @elmanu
110. THANK YOU
> Questions, comments, feedback?
> Contact me at manuel@bernhardt.io /
@elmanu
> Read more about Akka at
https://manuel.bernhardt.io/
articles
Webinar - https://manuel.bernhardt.io - @elmanu