1. Distributed by Design
Dr. Roland Kuhn
@rolandkuhn
onsdag 3 april 13

2. outline
actors
remote actors
rise of the cluster
when the cluster grows up
adding types
onsdag 3 april 13

3. What is an Actor?
• Akka's unit of computation is called an Actor
• Actors are purely reactive components:
– a mailbox
– behavior & state
– scheduled to run when sent a message
• Each actor has a parent, handling its failures
onsdag 3 april 13

4. Actor
Behavior
State
onsdag 3 april 13

5. Event-driven
Thread
Actor
Behavior
State
onsdag 3 april 13

6. Event-driven
Thread
Actor
Behavior
State
onsdag 3 april 13

7. Event-driven
Thread
Actor
Behavior
State
onsdag 3 april 13

8. Event-driven
Thread
Actor
Behavior
State
onsdag 3 april 13

9. Actor
Behavior
State
onsdag 3 april 13

10. Event-driven
Thread
Actor
Behavior
State
onsdag 3 april 13

11. Define Actor
public class Greeting implements Serializable {
public final String who;
public Greeting(String who) { this.who = who; }
}
public class GreetingActor extends UntypedActor {
LoggingAdapter log = Logging.getLogger(getContext().system(), this);
int counter = 0;
public void onReceive(Object message) {
if (message instanceof Greeting) {
counter++;
log.info("Hello #" + counter + " " + ((Greeting) message).who);
}
}
}
onsdag 3 april 13

12. Define Actor
Define the message(s) the Actor
should be able to respond to
public class Greeting implements Serializable {
public final String who;
public Greeting(String who) { this.who = who; }
}
public class GreetingActor extends UntypedActor {
LoggingAdapter log = Logging.getLogger(getContext().system(), this);
int counter = 0;
public void onReceive(Object message) {
if (message instanceof Greeting) {
counter++;
log.info("Hello #" + counter + " " + ((Greeting) message).who);
}
}
}
onsdag 3 april 13

13. Define Actor
Define the message(s) the Actor
should be able to respond to
public class Greeting implements Serializable {
public final String who;
public Greeting(String who)the Actor class = who; }
Define { this.who
}
public class GreetingActor extends UntypedActor {
LoggingAdapter log = Logging.getLogger(getContext().system(), this);
int counter = 0;
public void onReceive(Object message) {
if (message instanceof Greeting) {
counter++;
log.info("Hello #" + counter + " " + ((Greeting) message).who);
}
}
}
onsdag 3 april 13

14. Define Actor
Define the message(s) the Actor
should be able to respond to
public class Greeting implements Serializable {
public final String who;
public Greeting(String who)the Actor class = who; }
Define { this.who
}
public class GreetingActor extends UntypedActor {
LoggingAdapter log = Logging.getLogger(getContext().system(), this);
int counter = 0;
public void onReceive(Object message) {
if (message instanceof Greeting) {
counter++;
log.info("Hello #" + counter + " " + ((Greeting) message).who);
}
} Define the Actor’s behavior
}
onsdag 3 april 13

15. Create Actor
public class Greeting implements Serializable {
public final String who;
public Greeting(String who) { this.who = who; }
}
public class GreetingActor extends UntypedActor {
LoggingAdapter log = Logging.getLogger(getContext().system(), this);
int counter = 0;
public void onReceive(Object message) {
if (message instanceof Greeting) {
counter++;
log.info("Hello #" + counter + " " + ((Greeting) message).who);
}
}
}
ActorSystem system = ActorSystem.create("MySystem");
ActorRef greeter = system.actorOf(new Props(GreetingActor.class), "greeter");
onsdag 3 april 13

16. Create Actor
public class Greeting implements Serializable {
public final String who;
public Greeting(String who) { this.who = who; }
}
public class GreetingActor extends UntypedActor {
LoggingAdapter log = Logging.getLogger(getContext().system(), this);
int counter = 0;
public void onReceive(Object message) {
if (message instanceof Greeting) {
counter++;
log.info("Hello #" + counter + " " + ((Greeting) message).who);
} Create an Actor system
}
}
ActorSystem system = ActorSystem.create("MySystem");
ActorRef greeter = system.actorOf(new Props(GreetingActor.class), "greeter");
onsdag 3 april 13

17. Create Actor
public class Greeting implements Serializable {
public final String who;
public Greeting(String who) { this.who = who; }
}
public class GreetingActor extends UntypedActor {
LoggingAdapter log = Logging.getLogger(getContext().system(), this);
int counter = 0;
public void onReceive(Object message) {
if (message instanceof Greeting) {
counter++;
log.info("Hello #" + counter + " " + ((Greeting) message).who);
Actor configuration
} Create an Actor system
}
}
ActorSystem system = ActorSystem.create("MySystem");
ActorRef greeter = system.actorOf(new Props(GreetingActor.class), "greeter");
onsdag 3 april 13

18. Create Actor
public class Greeting implements Serializable {
public final String who;
public Greeting(String who) { this.who = who; }
}
public class GreetingActor extends UntypedActor {
LoggingAdapter log = Logging.getLogger(getContext().system(), this);
int counter = 0;
public void onReceive(Object message) {
if (message instanceof Greeting) {
counter++;
log.info("Hello #" + counter + " " + ((Greeting) message).who);
Actor configuration
} Create an Actor system
}
}
ActorSystem system = ActorSystem.create("MySystem");
ActorRef greeter = system.actorOf(new Props(GreetingActor.class), "greeter");
Give it a name
onsdag 3 april 13

19. Create Actor
public class Greeting implements Serializable {
public final String who;
public Greeting(String who) { this.who = who; }
}
public class GreetingActor extends UntypedActor {
LoggingAdapter log = Logging.getLogger(getContext().system(), this);
int counter = 0;
public void onReceive(Object message) {
if (message instanceof Greeting) {
counter++;
log.info("Hello #" + counter + " " + ((Greeting) message).who);
Actor configuration
} Create an Actor system
}
}
ActorSystem system = ActorSystem.create("MySystem");
ActorRef greeter = system.actorOf(new Props(GreetingActor.class), "greeter");
Create the Actor Give it a name
onsdag 3 april 13

20. Create Actor
public class Greeting implements Serializable {
public final String who;
public Greeting(String who) { this.who = who; }
}
public class GreetingActor extends UntypedActor {
LoggingAdapter log = Logging.getLogger(getContext().system(), this);
int counter = 0;
public void onReceive(Object message) {
if (message instanceof Greeting) {
counter++;
log.info("Hello #" + counter + " " + ((Greeting) message).who);
Actor configuration
} Create an Actor system
}
}
ActorSystem system = ActorSystem.create("MySystem");
ActorRef greeter = system.actorOf(new Props(GreetingActor.class), "greeter");
You get an ActorRef back Create the Actor Give it a name
onsdag 3 april 13

21. Actors can form hierarchies
Guardian System Actor
onsdag 3 april 13

22. Actors can form hierarchies
Guardian System Actor
system.actorOf(
new Props(Foo.class), “Foo”);
onsdag 3 april 13

23. Actors can form hierarchies
Guardian System Actor
system.actorOf(
Foo
new Props(Foo.class), “Foo”);
onsdag 3 april 13

24. Actors can form hierarchies
Guardian System Actor
Foo
getContext().actorOf(
new Props(A.class), “A”);
onsdag 3 april 13

25. Actors can form hierarchies
Guardian System Actor
Foo
getContext().actorOf(
A
new Props(A.class), “A”);
onsdag 3 april 13

26. Actors can form hierarchies
Guardian System Actor
Foo Bar
A C A
B E C
B
D
onsdag 3 april 13

27. Name resolution - like a file-system
Guardian System Actor
Foo Bar
A C A
B E C
B
D
onsdag 3 april 13

28. Name resolution - like a file-system
Guardian System Actor
/Foo
Foo Bar
A C A
B E C
B
D
onsdag 3 april 13

29. Name resolution - like a file-system
Guardian System Actor
/Foo
Foo Bar
/Foo/A
A C A
B E C
B
D
onsdag 3 april 13

30. Name resolution - like a file-system
Guardian System Actor
/Foo
Foo Bar
/Foo/A
A C A
/Foo/A/B
B E C
B
D
onsdag 3 april 13

31. Name resolution - like a file-system
Guardian System Actor
/Foo
Foo Bar
/Foo/A
A C A
/Foo/A/B
B E C
B
D /Foo/A/D
onsdag 3 april 13

32. Send Message
public class Greeting implements Serializable {
public final String who;
public Greeting(String who) { this.who = who; }
}
public class GreetingActor extends UntypedActor {
LoggingAdapter log = Logging.getLogger(getContext().system(), this);
int counter = 0;
public void onReceive(Object message) {
if (message instanceof Greeting) {
counter++;
log.info("Hello #" + counter + " " + ((Greeting) message).who);
}
}
}
ActorSystem system = ActorSystem.create("MySystem");
ActorRef greeter = system.actorOf(new Props(GreetingActor.class), "greeter");
greeter.tell(new Greeting("Charlie Parker"), null);
onsdag 3 april 13

33. Send Message
public class Greeting implements Serializable {
public final String who;
public Greeting(String who) { this.who = who; }
}
public class GreetingActor extends UntypedActor {
LoggingAdapter log = Logging.getLogger(getContext().system(), this);
int counter = 0;
public void onReceive(Object message) {
if (message instanceof Greeting) {
counter++;
log.info("Hello #" + counter + " " + ((Greeting) message).who);
}
}
}
ActorSystem system = ActorSystem.create("MySystem");
ActorRef greeter = system.actorOf(new Props(GreetingActor.class), "greeter");
greeter.tell(new Greeting("Charlie Parker"), null);
Send the message
onsdag 3 april 13

34. Full example
public class Greeting implements Serializable {
public final String who;
public Greeting(String who) { this.who = who; }
}
public class GreetingActor extends UntypedActor {
LoggingAdapter log = Logging.getLogger(getContext().system(), this);
int counter = 0;
public void onReceive(Object message) {
if (message instanceof Greeting) {
counter++;
log.info("Hello #" + counter + " " + ((Greeting) message).who);
}
}
}
ActorSystem system = ActorSystem.create("MySystem");
ActorRef greeter = system.actorOf(new Props(GreetingActor.class), "greeter");
greeter.tell(new Greeting("Charlie Parker"), null);
onsdag 3 april 13

35. Distributable
by Design
onsdag 3 april 13

36. onsdag 3 april 13

37. Error
Kernel
onsdag 3 april 13

38. Node 1 Node 2
Error
Kernel
onsdag 3 april 13

39. Create Actor
ActorRef greeter =
system.actorOf(new Props(
GreetingActor.class), "greeter");
onsdag 3 april 13

40. Remote Deployment
Just feed the ActorSystem with this configuration
akka {
actor {
provider = akka.remote.RemoteActorRefProvider
deployment {
/greeter {
remote =
}
}
}
}
onsdag 3 april 13

41. Remote Deployment
Just feed the ActorSystem with this configuration
Configure a Remote Provider
akka {
actor {
provider = akka.remote.RemoteActorRefProvider
deployment {
/greeter {
remote =
}
}
}
}
onsdag 3 april 13

42. Remote Deployment
Just feed the ActorSystem with this configuration
Configure a Remote Provider
akka {
For the Greeter actor {
actor
provider = akka.remote.RemoteActorRefProvider
deployment {
/greeter {
remote =
}
}
}
}
onsdag 3 april 13

43. Remote Deployment
Just feed the ActorSystem with this configuration
Configure a Remote Provider
akka {
For the Greeter actor {
actor
provider = akka.remote.RemoteActorRefProvider
deployment {
/greeter {
remote =
}
}
Define Remote Path
}
}
onsdag 3 april 13

44. Remote Deployment
Just feed the ActorSystem with this configuration
Configure a Remote Provider
akka {
For the Greeter actor {
actor
provider = akka.remote.RemoteActorRefProvider
deployment {
/greeter {
remote = akka://
}
}
Define Remote Path
} Protocol
}
onsdag 3 april 13

45. Remote Deployment
Just feed the ActorSystem with this configuration
Configure a Remote Provider
akka {
For the Greeter actor {
actor
provider = akka.remote.RemoteActorRefProvider
deployment {
/greeter {
remote = akka://MySystem
}
}
Define Remote Path
} Protocol Actor System
}
onsdag 3 april 13

46. Remote Deployment
Just feed the ActorSystem with this configuration
Configure a Remote Provider
akka {
For the Greeter actor {
actor
provider = akka.remote.RemoteActorRefProvider
deployment {
/greeter {
remote = akka://MySystem@machine1
}
}
Define Remote Path
} Protocol Actor System Hostname
}
onsdag 3 april 13

47. Remote Deployment
Just feed the ActorSystem with this configuration
Configure a Remote Provider
akka {
For the Greeter actor {
actor
provider = akka.remote.RemoteActorRefProvider
deployment {
/greeter {
remote = akka://MySystem@machine1:2552
}
}
Define Remote Path
} Protocol Actor System Hostname Port
}
onsdag 3 april 13

48. Remote Deployment
Just feed the ActorSystem with this configuration
Configure a Remote Provider
akka {
For the Greeter actor {
actor
provider = akka.remote.RemoteActorRefProvider
deployment {
/greeter {
remote = akka://MySystem@machine1:2552
}
}
Define Remote Path
} Protocol Actor System Hostname Port
}
Zero code changes
onsdag 3 april 13

49. Remote Lookup
ActorRef greeter =
system.actorFor(
"akka://MySystem@machine1:2552/user/greeter"
);
onsdag 3 april 13

50. Can you see the problem?
onsdag 3 april 13

51. Fixed Addresses
akka {
actor {
provider = akka.remote.RemoteActorRefProvider
deployment {
/greeter {
remote = akka://MySystem@machine1:2552
}
}
}
}
ActorRef greeter =
system.actorFor(
"akka://MySystem@machine1:2552/user/greeter"
);
onsdag 3 april 13

52. Akka Cluster
onsdag 3 april 13

53. Akka Cluster 2.1
• Gossip-based Cluster Membership
• Failure Detector
2 .1
• Cluster DeathWatch in
vi ew
• Cluster-Aware Routers p re
nt al
m e
ri
expe
er is
lu st
C
onsdag 3 april 13

54. Cluster Membership
• Node ring à la Riak / Dynamo
• Gossip-protocol for state dissemination
• Vector Clocks to detect convergence
onsdag 3 april 13

55. Node ring with gossiping members
Member
Node
onsdag 3 april 13

56. Node ring with gossiping members
Member
Node
Member Member
Node Node
Member Member
Node Node
Member Member
Node Node
Member Member
Node Node
Member
Node
onsdag 3 april 13

57. Node ring with gossiping members
Member
Node
Member Member
Node Node
Member Member
Node Node
Member Member
Node Node
Member Member
Node Node
Member
Node
onsdag 3 april 13

58. Node ring with gossiping members
Member
Node
Member Member
Node Node
Member Member
Node Node
Member Member
Node Node
Member Member
Node Node
Member
Node
onsdag 3 april 13

59. Vector Clock
• Vector Clocks are used to:
- Generate a partial ordering of events in
a distributed system
- Detecting causality violations
• We use Vector Clocks to to reconcile and
merge differences in cluster state
onsdag 3 april 13

60. Gossiping Protocol
onsdag 3 april 13

61. Gossiping Protocol
Used for:
onsdag 3 april 13

62. Gossiping Protocol
Used for:
– Cluster Membership
onsdag 3 april 13

63. Gossiping Protocol
Used for:
– Cluster Membership
– Configuration data
onsdag 3 april 13

64. Gossiping Protocol
Used for:
– Cluster Membership
– Configuration data
– Leader Determination
onsdag 3 april 13

65. Gossiping Protocol
Used for:
– Cluster Membership
– Configuration data
– Leader Determination
– Partitioning data
onsdag 3 april 13

66. Gossiping Protocol
Used for:
– Cluster Membership
– Configuration data
– Leader Determination
– Partitioning data
– Naming Service
onsdag 3 april 13

67. Push/Pull Gossip
• Push
2 .3+
in
– sender only sends versions (Vector Clock)
n
za tio
• Pull ti mi
op
– receivere
u r only asks for information for which it has
u toutdated version
f an
• Partly biased
– send fraction of gossip to nodes with older state
onsdag 3 april 13

68. Cluster Convergence
onsdag 3 april 13

69. Cluster Convergence
• When each Node has seen the same Vector Clock
onsdag 3 april 13

70. Cluster Convergence
• When each Node has seen the same Vector Clock
• unreachable nodes will fail this
onsdag 3 april 13

71. Cluster Convergence
• When each Node has seen the same Vector Clock
• unreachable nodes will fail this
• mark nodes DOWN to proceed
onsdag 3 april 13

72. Cluster Convergence
• When each Node has seen the same Vector Clock
• unreachable nodes will fail this
• mark nodes DOWN to proceed
– manual Ops intervention
onsdag 3 april 13

73. Cluster Convergence
• When each Node has seen the same Vector Clock
• unreachable nodes will fail this
• mark nodes DOWN to proceed
– manual Ops intervention
– automatic action
onsdag 3 april 13

74. Member States
onsdag 3 april 13

75. Member States
• JOINING
• UP
• LEAVING
• EXITING
• DOWN
• REMOVED
onsdag 3 april 13

76. Member States
• JOINING
• UP
• LEAVING
• EXITING
• DOWN
• REMOVED
onsdag 3 april 13

77. Leader
onsdag 3 april 13

78. Leader
• Any node can be the leader
onsdag 3 april 13

79. Leader
• Any node can be the leader
• Just takes the role of being a leader
onsdag 3 april 13

80. Leader
• Any node can be the leader
• Just takes the role of being a leader
• Is deterministically recognized by all nodes
onsdag 3 april 13

81. Leader
• Any node can be the leader
• Just takes the role of being a leader
• Is deterministically recognized by all nodes
– always the first member in the sorted
membership ring
onsdag 3 april 13

82. Cluster Events
public class Listener extends UntypedActor {
public void onReceive(Object message) {
if (message instanceof MemberUp) {
// ...
}
}
}
ActorRef listener =
system.actorOf(new Props(Listener.class),
"listener");
Cluster.get(system).subscribe(listener,
MemberEvent.class);
onsdag 3 april 13

83. Cluster Events
public class Listener extends UntypedActor {
public void onReceive(Object message) {
if (message instanceof MemberUp) {
MemberUp mUp = (MemberUp) message;
getContext().actorFor(mUp.address() +
"/user/greeter").tell(
new Greeting("Charlie Parker"), getSelf());
}
}
}
ActorRef listener =
system.actorOf(new Props(Listener.class),
"listener");
Cluster.get(system).subscribe(listener,
MemberEvent.class);
onsdag 3 april 13

84. Phi Accrual Failure Detector
regular messages
A B
• B monitors A
• Sample inter-arrival time to expect next beat
• B measures continuum of deadness of A
http://ddg.jaist.ac.jp/pub/HDY+04.pdf
onsdag 3 april 13

85. Phi Accrual Failure Detector
regular messages
A B
• B monitors A
• Sample inter-arrival time to expect next beat
• B measures continuum of deadness of A
http://ddg.jaist.ac.jp/pub/HDY+04.pdf
onsdag 3 april 13

86. Selective Failure Detection
Member
Node
Member Member
Node Node
Member Member
Node Node
Heartbeat
Member Member
Node Node
Member
Member
Node
Member Node
Node
onsdag 3 april 13

87. Selective Failure Detection
Member
Node
Member Member
Node Node
Member Member
Node Node
Heartbeat
Member Member
Node Node
Member
Member
Node
Member Node
Node
onsdag 3 april 13

88. Cluster DeathWatch
• Triggered by marking node «A» DOWN
– Tell parents of their lost children on «A»
– Kill all children of actors on «A»
– Send Terminated for actors on «A»
onsdag 3 april 13

89. Enable clustering
akka {
actor {
provider = "akka.cluster.ClusterActorRefProvider"
...
}
extensions = ["akka.cluster.Cluster"]
cluster {
seed-nodes = [
"akka://ClusterSystem@127.0.0.1:2551",
"akka://ClusterSystem@127.0.0.1:2552"
]
}
}
onsdag 3 april 13

90. Load Balancing
onsdag 3 april 13

91. Routers
ActorRef routerActor =
getContext().actorOf(
new Props(ExampleActor.class).
withRouter(new RoundRobinRouter(nrOfInstances))
);
onsdag 3 april 13

92. …or from config
akka.actor.deployment {
/path/to/actor {
router = round-robin
nr-of-instances = 5
}
}
onsdag 3 april 13

93. Configure a clustered router
akka.actor.deployment {
/statsService/workerRouter {
router = consistent-hashing
nr-of-instances = 100
cluster {
enabled = on
max-nr-of-instances-per-node = 3
allow-local-routees = on
}
}
}
onsdag 3 april 13

94. Multi Node Testing
object MultiNodeSampleConfig extends MultiNodeConfig {
val node1 = role("node1")
val node2 = role("node2")
}
"A MultiNodeSample" must {
"wait for all nodes to enter a barrier" in {
enterBarrier("startup")
}
}
onsdag 3 april 13

95. Multi Node Testing
runOn(node2) {
system.actorOf(Props(new Actor {
def receive = {
case "ping" => sender ! "pong"
}
}), "ponger")
}
enterBarrier("deployed")
runOn(node1) {
val ponger =
system.actorFor(node(node2) / "user" / "ponger")
ponger ! "ping"
expectMsg("pong")
}
enterBarrier("finished")
onsdag 3 april 13

96. … when the Cluster grows up
onsdag 3 april 13

97. Adaptive Load Balancing
• Metrics collected and spread
– Heap memory
– CPU, system load
• Adaptive Router
– Biased random with weights based on capacity
onsdag 3 april 13

98. One tree to rule them all
• One Actor tree per node
• Cluster tree is mapped to local sub-trees
onsdag 3 april 13

99. One tree to rule them all
onsdag 3 april 13

100. One tree to rule them all
onsdag 3 april 13

101. One tree to rule them all
onsdag 3 april 13

102. The Magic Sauce
• User code only sees cluster://... names
• ActorRef becomes repointable
– local
– remote
• Can now move actors around transparently
– Actor encapsulation makes it possible
onsdag 3 april 13

103. What does this enable?
• Actor migration
• Actor replication
• Automatic cluster partitioning
– later also based on runtime metrics
• Node fail-over
– first for stateless actors
– later for stateful actors using event sourcing
➾ Fault Tolerance & Distribution
onsdag 3 april 13

104. Typed Channels
Experimental in 2.2
onsdag 3 april 13

105. The Problem
someActor ! CommandOne
onsdag 3 april 13

106. The Problem
trait Command
case class CommandOne(param: String) extends Command
someActor ! CommandOne
onsdag 3 april 13

107. The Vision
someActor <-!- CommandOne(”msg”)
because the other does not compile
onsdag 3 april 13

108. But How?
• ActorRef must know about message types
– Actor type must be parameterized
• Message type is verified against that
onsdag 3 april 13

109. And the replies?
val f: Future[Response] =
someActor <-?- CommandOne(”hello”)
because the compiler knows
onsdag 3 april 13

110. And How This?
• ActorRef must know reply types
– Actor must be parameterized with them
• Reply types are extracted at call site
onsdag 3 april 13

111. No Type Pollution
• Generic Filter/Transform Actors
– accept management commands
– pass on generic other type
• Using just one type is not enough!
• Need to use type unions and allow multiple
possible reply types for one input
onsdag 3 april 13

112. The Implementation
• Tagged type union with
:+:[(In, Out), ChannelList] <: ChannelList
• Value class ChannelRef[…](val a: ActorRef)
• Actor mixin Channels[…]
• WrappedMessage[…, LUB](val m: LUB)
• ops desugar to tell/ask after type check
onsdag 3 april 13

113. Actors Do Compose
msg -?-> firstActor -?-> secondActor -!-> client
msg -?-> someService -*-> (_ map httpOk) -!-> client
Process wiring from the outside
onsdag 3 april 13

114. How to Declare it?
class OpinionatedEcho extends Actor
with Channels[TNil, (String, String) :+: TNil] {
channel[String] { (str, sender) ⇒ sender <-!- str }
// or
channel[String] {
case (”hello”, sender) ⇒ sender <-!- ”world”
case (x, sender) ⇒ sender <-!- s”dunno: $x”
}
}
“sender” will accept only String messages
onsdag 3 april 13

115. The Result:
onsdag 3 april 13

116. Type-Safe Composability
of
Actor Systems
onsdag 3 april 13

117. get it and learn more
http://akka.io
http://letitcrash.com
http://typesafe.com
onsdag 3 april 13

118. E0F
onsdag 3 april 13