Downloaded 66 times
![case class Gossip(
members: immutable.SortedSet[Member], // sorted set of members with their status, sorted by address
overview: GossipOverview = GossipOverview(),
version: VectorClock = VectorClock()) // vector clock version
case class GossipOverview(
seen: Set[UniqueAddress] = Set.empty,
reachability: Reachability = Reachability.empty)
case class VectorClock(
versions: TreeMap[VectorClock.Node, Long] = TreeMap.empty[VectorClock.Node, Long]) {
/**
* Compare two vector clocks. The outcome will be one of the following:
* <p/>
* {{{
* 1. Clock 1 is SAME (==) as Clock 2 iff for all i c1(i) == c2(i)
* 2. Clock 1 is BEFORE (<) Clock 2 iff for all i c1(i) <= c2(i)
* and there exist a j such that c1(j) < c2(j)
* 3. Clock 1 is AFTER (>) Clock 2 iff for all i c1(i) >= c2(i)
* and there exist a j such that c1(j) > c2(j).
* 4. Clock 1 is CONCURRENT (<>) to Clock 2 otherwise.
* }}}
*/
def compareTo(that: VectorClock): Ordering = {
compareOnlyTo(that, FullOrder)
}
}](https://image.slidesharecdn.com/meetupakkacluster-160826152623/85/Akka-Cluster-in-Production-11-320.jpg)
More Related Content
Similar to Akka Cluster in Production
![JavaOne: A tour of (advanced) akka features in 60 minutes [con1706]](https://cdn.slidesharecdn.com/ss_thumbnails/atourofadvancedakkafeaturesin60minutescon1706-160921210355-thumbnail.jpg?width=640&height=640&fit=bounds)
![5G Explained! A High Level Overview [Introduction]](https://cdn.slidesharecdn.com/ss_thumbnails/5gexplainedahighleveloverview-260119165306-cc137a3e-thumbnail.jpg?width=640&height=640&fit=bounds)
Akka Cluster in Production
- 1. Akka Cluster inProduction
- 2. • Reactive? • Actors? •Akka? • Clustering? • Onlineshopping? • Online payment?
- 3. Device Ident • Detectfraudulent devices in real time • Retail, telco, finance, … • Integrated via snippet into webshop • Analyses client devices • Results obtained via REST API • > 300 sites, • > 100M devices • 24/7, 3 nines
- 4.
- 5. Actors „The actor modelin computer science is a mathematical model of concurrent computation that treats "actors" as the universal primitives of concurrent computation. In response to a message that it receives, an actor can: • make local decisions, • create more actors, • send more messages, and • determine how to respond to the next message received. Actors may modify private state, but can only affect each other through messages (avoiding the need for any locks).“ https://en.wikipedia.org/wiki/Actor_model
- 6. Akka Cluster Components Remoting Clustering Distributed PubSub ClusterSingleton Sharding Distributed Data Persistence Cluster Client Cluster Aware Routers Actors
- 7. Akka Remote • replaceLocalActorRefProvider by RemoteActorRefProvider • ActorRef: akka://systemName/user/parent/actorName • Remote ActorRef: akka.tcp://systemName@hostName:1234/user/parent/actorName • Look up remote actors • Start remote actors • Cluster aware routing • death watch
- 8. Failure Detector • ThePhi Accrual Failure Detector phi = -log10(1 - F(timeSinceLastHeartbeat)) • Each node monitored by small number of other nodes determined using Hash Ring • Output: confidence that a node is unreachable • Also notices when it becomes reachable again
- 9. Akka Cluster • ClusterMembership managed using Gossip Protocol • Dynamo based system • Subscribe to cluster state events • Roles • Restrictions on number of nodes possible (also per role) • When Gossip Convergence is reached, a leader can deterministically be determined • Head of list of nodes in alphanumeric order • Leader joins / removes members • Leader can auto-down members • Join manually or to Seed Nodes • The first seed node joins itself
- 10.
- 11. case class Gossip( members:immutable.SortedSet[Member], // sorted set of members with their status, sorted by address overview: GossipOverview = GossipOverview(), version: VectorClock = VectorClock()) // vector clock version case class GossipOverview( seen: Set[UniqueAddress] = Set.empty, reachability: Reachability = Reachability.empty) case class VectorClock( versions: TreeMap[VectorClock.Node, Long] = TreeMap.empty[VectorClock.Node, Long]) { /** * Compare two vector clocks. The outcome will be one of the following: * <p/> * {{{ * 1. Clock 1 is SAME (==) as Clock 2 iff for all i c1(i) == c2(i) * 2. Clock 1 is BEFORE (<) Clock 2 iff for all i c1(i) <= c2(i) * and there exist a j such that c1(j) < c2(j) * 3. Clock 1 is AFTER (>) Clock 2 iff for all i c1(i) >= c2(i) * and there exist a j such that c1(j) > c2(j). * 4. Clock 1 is CONCURRENT (<>) to Clock 2 otherwise. * }}} */ def compareTo(that: VectorClock): Ordering = { compareOnlyTo(that, FullOrder) } }
- 12.
- 13. Cluster Singletons • e.g.single point of entry, centralized routing logic, … • live on the oldest node • ClusterSingletonManager started on each node • ClusterSingletonProxy for accessing current Singleton
- 14. Cluster Singleton Singleton Manager Singleton Proxy Singleton Manager Node1 Node2Node3 Singleton Proxy Singleton Proxy Singleton
- 15. Cluster Singleton Singleton Manager Singleton Proxy Singleton Manager Node1 Node2Node3 Singleton Proxy Singleton Proxy Singleton
- 16. Cluster Singletons • e.g.single point of entry, centralized routing logic, … • live on the oldest node • ClusterSingletonManager started on each node • ClusterSingletonProxy for accessing current Singleton • caveats: • Single point of bottleneck • Must recover state on migration • In case of split brain, multiple singletons
- 17. Distributed PubSub • DistributedPubSubMediatorstarted on all nodes • Subscriptions are gossiped, eventually consistent • modes Publish, Group Publish, Send • used e.g. for cluster wide config, chat system, …
- 18. Cluster Sharding • DistributeWork • Workload partitioned by shard key derived from message • Messages must be serializable • Each node is responsible for n shards and each shard is allocated to one node • ShardRegion is entry point for messages and controls workers • ShardCoordinator singleton assigns shards • Shards distributed by no of workers by default • Shards migrate for rebalancing or on failure • Shard assignments can be persisted • Running Workers per shard can be remembered. • workers must step down • Workers must persist state if they need it after migration
- 19. Cluster Sharding Shard Region Shard Coordinator Shard Region Shard Region Proxy Node1Node2 Node3 Shard NodeShard Node Shard Node Shard Node
- 20. Cluster Sharding Shard Region Shard Coordinator Shard Region Shard Region Proxy Node1Node2 Node3 Shard NodeShard Node Shard Node Shard Node ! „Hello World“ key = 1234 shard = 12
- 21. Cluster Sharding Shard Region Shard Coordinator Shard Region Shard Region Proxy Node1Node2 Node3 Shard NodeShard Node Shard Node Shard Node ! „Hello World“ key = 1234 shard = 12 ? 12
- 22. Cluster Sharding Shard Region Shard Coordinator Shard Region Shard Region Proxy Node1Node2 Node3 Shard NodeShard Node Shard Node 12 Node1 Shard Node ! „Hello World“ key = 1234 shard = 12 ! Node1
- 23. Cluster Sharding Shard Region Shard Coordinator Shard Region Shard Region Proxy Node1Node2 Node3 Shard NodeShard Node 12 Node1 Shard Node 12 Node1 Shard Node Entity 1234
- 24. Cluster Sharding Shard Region Shard Coordinator Shard Region Shard Region Proxy Node1Node2 Node3 Shard NodeShard Node 12 Node1 Shard Node 12 Node1 Shard Node Entity 1234 !„HelloWorld
- 25. Cluster Sharding Shard Region Shard Coordinator Shard Region Shard Region Proxy Node1Node2 Node3 Shard NodeShard Node 12 Node1 Shard Node 12 Node1 Shard Node Entity 1234 ! „Hello World“ key = 1234 shard = 12 ? 12
- 26. Cluster Sharding Shard Region Shard Coordinator Shard Region Shard Region Proxy Node1Node2 Node3 Shard NodeShard Node 12 Node1 Shard Node 12 Node1 Shard Node Entity 1234 ! „Hello World“ key = 1234 shard = 12 ! Node1
- 27. Cluster Sharding Shard Region Shard Coordinator Shard Region Shard Region Proxy Node1Node2 Node3 Shard Node 12 Node1 Shard Node 12 Node1 Shard Node 12 Node1 Shard Node Entity 1234 ! „Hello World“ key = 1234 shard = 12 ! Node1
- 28. Cluster Sharding Shard Region Shard Coordinator Shard Region Shard Region Proxy Node1Node2 Node3 Shard Node 12 Node1 Shard Node 12 Node1 Shard Node 12 Node1 Shard Node Entity 1234 ! „Hello World
- 29. Cluster Sharding Shard Region Shard Coordinator Shard Region Shard Region Proxy Node1Node2 Node3 Shard Node 12 Node1 Shard Node 12 Node1 Shard Node 12 Node1 Shard Node Entity 1234 ! „Happy Day“ key = 1299 shard = 12
- 30. Cluster Sharding Shard Region Shard Coordinator Shard Region Shard Region Proxy Node1Node2 Node3 Shard Node 12 Node1 Shard Node 12 Node1 Shard Node 12 Node1 Shard Node Entity 1234 Entity 1299 ! „Happy Day“
- 31. Cluster Sharding Shard Region Shard Coordinator Shard Region Shard Region Proxy Node1Node2 Node3 Shard Node 12 Node1 9 Node2 Shard Node 12 Node1 9 Node2 3 Node1 Shard Node 12 Node1 9 Node2 3 Node1 Shard Node 9 Node2 3 Node1 Entity 1234 Entity 0902 Entity 0901 Entity 345 Entity 1299
- 32. Cluster Sharding Shard Region Shard Coordinator Shard Region Shard Region Proxy Node1Node2 Node3 Shard Node 9 Node2 Shard Node 9 Node2 Entity 1234 Entity 0902 Entity 0901 Entity 345 Entity 1299 Shard Coordinator Shard Node Not remembering shards
- 33. Cluster Sharding Shard Region Shard Coordinator Shard Region Shard Region Proxy Node1Node2 Node3 Shard Node 9 Node2 Shard Node 9 Node2 Entity 1234 Entity 0902 Entity 0901 Entity 345 Entity 1299 Shard Coordinator Shard Node 9 Node2 Not remembering shards / ddata https://github.com/akka/akka/issues/19003
- 34. Cluster Sharding Shard Region Shard Coordinator Shard Region Shard Region Proxy Node1Node2 Node3 Shard Node 9 Node2 Shard Node Entity 1234 Entity 0902 Entity 0901 Entity 345 Entity 1299 Shard Coordinator Shard Node 12 Node2 9 Node2 3 Node2 Entity 1234 Entity 345 Entity 1299 remembering shards / persistence
- 35. Sharding and Persistence •Persist ShardCoordinator state • akka.cluster.sharding { state-store-mode = „persistence" journal-plugin-id = "akka-contrib-mongodb-persistence-journal-sharding" snapshot-plugin-id = "akka-contrib-mongodb-persistence-snapshot-sharding" } • akka.cluster.sharding.state-store-mode = "ddata" • Remember Shard entities • akka.cluster.sharding.remember-entities = "on" • Step down of workers • super ! Passivate(StopMessage)
- 36. Distributed Data • KV-storebased on Conflict Free Replicated Data Types (CRDTs) • Counters: GCounter, PNCounter • Sets: GSet, ORSet • Maps: ORMap, ORMultiMap, LWWMap, PNCounterMap • Registers: LWWRegister, Flag • not intended for Big Data: In memory, full state replicated • start ddata.Replicator val Counter1Key = PNCounterKey("counter1") replicator ! Update(Counter1Key, PNCounter(), WriteLocal)(_ + 1) val readFrom3 = ReadFrom(n = 3, timeout = 1.second) replicator ! Get(Counter1Key, readFrom3) • Consistency levels • ReadLocal / WriteLocal • ReadFromN / WriteToN • ReadMajority / WriteMajority • WriteAll and ReadAll • Can subscribe to changes
- 37.
- 38.
- 39.
- 40.
- 41.
- 42. Live Example... • config •scaling horizontally • cluster client • multi JVM test • singleton monitoring throughput and lifecycle
- 43. Caveats and Lessonslearned • Remoting Setup • TLS certificates rolling update • difficult to test if new settings work whole the old ones are still there • In our case: export restricted crypto • Not too critical, if noticed early in rolling upgrade • Adjust Failure detector settings to your environment • quite strict for us • must accept higher latencies & short interruptions in cloud environment • Configure internal & external hostname in containerized / NATed / … environment • hostname and IP are completely different for Akka!
- 44. Caveats and Lessonslearned • Cluster Setup • Currently rather static hardware environment • Joining to list of seed nodes. • Had a split brain once, need to carefully restart the right part of the cluster • Log cluster state each node sees (or use JMX)! • preventing split brain • maybe disable auto down • split brain resolver • adjust failure detector settings to your environment • restart order youngest to oldest to minimize singleton migrations
- 45. Caveats and Lessonslearned • Sharding • Recovery of persistent actors should be planned (might take time) • If shard coordinator fails to recover you're doomed • separate journal for internal sharding state • can be cleaned if cluster is shutdown • ICE delete journal • Will allow for ShardCoordinator recovery • Will make shard allocation state in ShardRegions inconsistent • Fix state by rolling restart
- 46. Shutdown // Play cannot stop accepting requests. // Fail healthcheck, so the loadbalancer removes this node. GlobalHealthcheck.fail(reason = shuttingDown) // migrate all shards to other nodes, stop accepting new ones val cluster = Cluster(context.system) context.watch(region) region ! ShardRegion.GracefulShutdown // After shutdown of ShardRegion, shutdown the ActorSystem case Terminated(`region`) => cluster.leave(cluster.selfAddress) cluster.registerOnMemberRemoved { system.terminate } // After shutdown of the ActorSystem, shutdown the App system.registerOnTermination { System.exit(0) }
- 47.