Dr. Roland Kuhn
Akka Tech Lead
@rolandkuhn
Distributed in Space and Time
The Future is Bright
2
source: http://nature.desktopnexus.com/wallpaper/58502/
The Future is Bright
• Computing Resources: Effectively Unbounded
• 18-core CPU packages, 1024 around the corner
• on-dema...
How do I make a Large-Scale App?
• figure out a service to provide to everyone
• this will require resources accordingly:
...
How do I make an Always-Available App?
• must not have single point of failure
• all functions and resources are replicate...
How do I make a Persistent App?
• runtime replication usually not sufficient
• system of record needed
• one central datab...
How do I make a Persistent App?
7
How do I make a Persistent App?
8
How do I make a Persistent App?
• runtime replication usually not sufficient
• system of record needed
• one central datab...
As Noted Earlier: The Future is Bright
10
source: http://nature.desktopnexus.com/wallpaper/58502/
… or is it?
11
Photograph: Patrick Cromerford, 2011
Distribution: a Double-Edged Sword
• did that message make it over the network?
• is that other machine running?
• are our...
The Essence of the Problem
A distributed system is one whose parts can fail
independently of each other.
13
Fighting Back!
14
source: www.hear2heal.com (Kokopelli Rain Dance)
Transactional Storage
Consensus
Message Delivery
…
Transactional Storage
• Serializability simplifies reasoning about behavior
• … but is at odds with distribution (CAP theo...
PAXOS and Friends
• problem: distributed consensus
• hard solutions favor Consistency
• 2PC, PAXOS, RAFT
• if a node is un...
Fighting Message Loss
• resending is the only cure
• sender stores the message and retries
• recipient must send acknowled...
At-Least-Once Delivery with Akka
19
class ALOD(other: ActorPath) extends PersistentActor with AtLeastOnceDelivery {
val ID...
At-Least-Once Delivery with Akka
20
class ALOD(other: ActorPath) extends PersistentActor with AtLeastOnceDelivery {
val ID...
At-Least-Once Delivery with Akka
21
class ALOD(other: ActorPath) extends PersistentActor with AtLeastOnceDelivery {
val ID...
At-Least-Once Delivery with Akka
22
class ALOD(other: ActorPath) extends PersistentActor with AtLeastOnceDelivery {
val ID...
At-Least-Once Delivery with Akka
23
class ALOD(other: ActorPath) extends PersistentActor with AtLeastOnceDelivery {
val ID...
Mysterious Double-Bookings
• acknowledgements can be lost
• sender will faithfully duplicate the message
• therefore named...
Exactly-Once Delivery
• processing occurs only once for each message
• all middleware does it*
• EIP: Guaranteed Delivery ...
Exactly-Once Delivery with Akka
26
class PersistThenAct extends PersistentActor {
var nextId = 1
def receiveCommand = {
ca...
Exactly-Once Delivery with Akka
27
class PersistThenAct extends PersistentActor {
var nextId = 1
def receiveCommand = {
ca...
Exactly-Once Delivery with Akka
28
class PersistThenAct extends PersistentActor {
var nextId = 1
def receiveCommand = {
ca...
Exactly-Once Delivery with Akka
29
class PersistThenAct extends PersistentActor {
var nextId = 1
def receiveCommand = {
ca...
Exactly-Once Delivery with Akka
30
class PersistThenAct extends PersistentActor {
var nextId = 1
def receiveCommand = {
ca...
Exactly-Once Delivery with Akka
31
class PersistThenAct extends PersistentActor {
var nextId = 1
def receiveCommand = {
ca...
The Hard Truth
CPU can stop between any two instructions, therefore
exactly-once semantics cannot be Guaranteed*.
*with a ...
Mitigating that Hard Truth
• probability of loss or duplication can be small
• many systems are fine with almost-exactly-o...
We Can Do Better!
• an operation is idempotent if multiple application
produces the same result as single application
• (s...
Effectively-Exactly-Once with Akka
35
class Idempotent(other: ActorRef) extends PersistentActor {
var nextId = 1
def recei...
The Future is Bright Again!
36
source: http://freewallpapersbackgrounds.com/
To Recapitulate:
37
Problem Choices
Transactional Storage
monolithic or distributed/partitioned,
serializable or eventuall...
Conclusion: Consistency à la Carte
• not all parts of a system have the same needs
• expend the effort only where necessar...
Remember that the trade-off is always between
consistency and availability.
And latency.
©Typesafe 2014 – All Rights Reserved
Upcoming SlideShare
Loading in …5
×

Distributed in Space and Time

1,555 views

Published on

How do I create an application that is always available, even in the face of catastrophic hardware failures? How do I make it cope with large fluctuations in the number of active users or their behavior? The answers to both questions have one thing in common: the need to spread the application out, to distribute it. In this presentation we take a deep dive into the architecture of reactive applications (see http://reactivemanifesto.org/), starting from cheap and efficient best-effort messaging and building delivery guarantees and persistence on top—with examples using Akka. Understanding these mechanisms and how they relate to each other will allow you to build systems that strike the right balance between partition tolerance, consistency, availability, implementation complexity and operational cost.

Published in: Software
0 Comments
9 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,555
On SlideShare
0
From Embeds
0
Number of Embeds
65
Actions
Shares
0
Downloads
30
Comments
0
Likes
9
Embeds 0
No embeds

No notes for slide

Distributed in Space and Time

  1. 1. Dr. Roland Kuhn Akka Tech Lead @rolandkuhn Distributed in Space and Time
  2. 2. The Future is Bright 2 source: http://nature.desktopnexus.com/wallpaper/58502/
  3. 3. The Future is Bright • Computing Resources: Effectively Unbounded • 18-core CPU packages, 1024 around the corner • on-demand provisioning in the cloud • Demand for Web-Scale Apps: Ravenous • everyone has the tools • successful ideas are bought for Giga-$ 3
  4. 4. How do I make a Large-Scale App? • figure out a service to provide to everyone • this will require resources accordingly: • storage • computation • communication • scalability implies no shared contention points • need to be elastic ➟ use the cloud 4
  5. 5. How do I make an Always-Available App? • must not have single point of failure • all functions and resources are replicated • protect against data loss • protect against service downtime • regional content delivery • need to distribute ➟ use the cloud 5
  6. 6. How do I make a Persistent App? • runtime replication usually not sufficient • system of record needed • one central database is not tenable • persist “locally” and replicate changes 6
  7. 7. How do I make a Persistent App? 7
  8. 8. How do I make a Persistent App? 8
  9. 9. How do I make a Persistent App? • runtime replication usually not sufficient • system of record needed • one central database is not tenable • persist “locally” and replicate changes • temporally distributed change records • spatially distributed change log • granularity can be one Aggregate Root (Event Sourcing) • extract business intelligence from history • scale & distribute persistence ➟ in the cloud 9
  10. 10. As Noted Earlier: The Future is Bright 10 source: http://nature.desktopnexus.com/wallpaper/58502/
  11. 11. … or is it? 11 Photograph: Patrick Cromerford, 2011
  12. 12. Distribution: a Double-Edged Sword • did that message make it over the network? • is that other machine running? • are our data centers in sync? • did we lose messages in that hardware crash? 12
  13. 13. The Essence of the Problem A distributed system is one whose parts can fail independently of each other. 13
  14. 14. Fighting Back! 14 source: www.hear2heal.com (Kokopelli Rain Dance)
  15. 15. Transactional Storage Consensus Message Delivery …
  16. 16. Transactional Storage • Serializability simplifies reasoning about behavior • … but is at odds with distribution (CAP theorem) • Eventual Consistency is often sufficient
 (PeterBailis:ProbabilisticallyBoundedStaleness) • Scalability can be regained by partitioning
 (PatHelland:LifebeyondDistributedTransactions) • consistency has a cost • you need to choose how much is necessary 16
  17. 17. PAXOS and Friends • problem: distributed consensus • hard solutions favor Consistency • 2PC, PAXOS, RAFT • if a node is unreachable then consensus cannot be reached • soft solutions favor Availability • quorum, allow nodes to drop out • partitions can lead to more than one active set • consistency always has a cost • you need to choose which one is appropriate 17
  18. 18. Fighting Message Loss • resending is the only cure • sender stores the message and retries • recipient must send acknowledgement eventually • fighting unreliable medium is sender’s obligation • sender’s buffer must be persistent • resending must begin anew after a crash 18
  19. 19. At-Least-Once Delivery with Akka 19 class ALOD(other: ActorPath) extends PersistentActor with AtLeastOnceDelivery { val IDs = Iterator from 1 def receiveCommand = { case Command(x) => persist(ImportantResult(x)) { ir => sender() ! Done(x) deliver(other, FollowUpCommand(x, IDs.next(), _)) } case rc: ReceiptConfirmed => persist(rc) { rc => confirmDelivery(rc.deliveryId) } } def receiveRecover = { case ImportantResult(x) => deliver(other, FollowUpCommand(x, IDs.next(), _)) case ReceiptConfirmed(_, id) => confirmDelivery(id) } }
  20. 20. At-Least-Once Delivery with Akka 20 class ALOD(other: ActorPath) extends PersistentActor with AtLeastOnceDelivery { val IDs = Iterator from 1 def receiveCommand = { case Command(x) => persist(ImportantResult(x)) { ir => sender() ! Done(x) deliver(other, FollowUpCommand(x, IDs.next(), _)) } case rc: ReceiptConfirmed => persist(rc) { rc => confirmDelivery(rc.deliveryId) } } def receiveRecover = { case ImportantResult(x) => deliver(other, FollowUpCommand(x, IDs.next(), _)) case ReceiptConfirmed(_, id) => confirmDelivery(id) } }
  21. 21. At-Least-Once Delivery with Akka 21 class ALOD(other: ActorPath) extends PersistentActor with AtLeastOnceDelivery { val IDs = Iterator from 1 def receiveCommand = { case Command(x) => persist(ImportantResult(x)) { ir => sender() ! Done(x) deliver(other, FollowUpCommand(x, IDs.next(), _)) } case rc: ReceiptConfirmed => persist(rc) { rc => confirmDelivery(rc.deliveryId) } } def receiveRecover = { case ImportantResult(x) => deliver(other, FollowUpCommand(x, IDs.next(), _)) case ReceiptConfirmed(_, id) => confirmDelivery(id) } }
  22. 22. At-Least-Once Delivery with Akka 22 class ALOD(other: ActorPath) extends PersistentActor with AtLeastOnceDelivery { val IDs = Iterator from 1 def receiveCommand = { case Command(x) => persist(ImportantResult(x)) { ir => sender() ! Done(x) deliver(other, FollowUpCommand(x, IDs.next(), _)) } case rc: ReceiptConfirmed => persist(rc) { rc => confirmDelivery(rc.deliveryId) } } def receiveRecover = { case ImportantResult(x) => deliver(other, FollowUpCommand(x, IDs.next(), _)) case ReceiptConfirmed(_, id) => confirmDelivery(id) } }
  23. 23. At-Least-Once Delivery with Akka 23 class ALOD(other: ActorPath) extends PersistentActor with AtLeastOnceDelivery { val IDs = Iterator from 1 def receiveCommand = { case Command(x) => persist(ImportantResult(x)) { ir => sender() ! Done(x) deliver(other, FollowUpCommand(x, IDs.next(), _)) } case rc: ReceiptConfirmed => persist(rc) { rc => confirmDelivery(rc.deliveryId) } } def receiveRecover = { case ImportantResult(x) => deliver(other, FollowUpCommand(x, IDs.next(), _)) case ReceiptConfirmed(_, id) => confirmDelivery(id) } }
  24. 24. Mysterious Double-Bookings • acknowledgements can be lost • sender will faithfully duplicate the message • therefore named at-least-once • recipient is responsible for dealing with this 24
  25. 25. Exactly-Once Delivery • processing occurs only once for each message • all middleware does it* • EIP: Guaranteed Delivery just means persistence • JMS: configure session for AUTO_ACKNOWLEDGE • Google’s MillWheel 25
  26. 26. Exactly-Once Delivery with Akka 26 class PersistThenAct extends PersistentActor { var nextId = 1 def receiveCommand = { case FollowUpCommand(x, id, deliveryId) => val rc = ReceiptConfirmed(id, deliveryId) if (id == nextId) { persist(rc) { rc => sender() ! rc doTheAction() nextId += 1 } } else if (id < nextId) sender() ! rc // otherwise an older command was lost, so wait for the resend } def receiveRecover = { case ReceiptConfirmed(id, _) => nextId = id + 1 } private def doTheAction(): Unit = () // actually do the action }
  27. 27. Exactly-Once Delivery with Akka 27 class PersistThenAct extends PersistentActor { var nextId = 1 def receiveCommand = { case FollowUpCommand(x, id, deliveryId) => val rc = ReceiptConfirmed(id, deliveryId) if (id == nextId) { persist(rc) { rc => sender() ! rc doTheAction() nextId += 1 } } else if (id < nextId) sender() ! rc // otherwise an older command was lost, so wait for the resend } def receiveRecover = { case ReceiptConfirmed(id, _) => nextId = id + 1 } private def doTheAction(): Unit = () // actually do the action }
  28. 28. Exactly-Once Delivery with Akka 28 class PersistThenAct extends PersistentActor { var nextId = 1 def receiveCommand = { case FollowUpCommand(x, id, deliveryId) => val rc = ReceiptConfirmed(id, deliveryId) if (id == nextId) { persist(rc) { rc => sender() ! rc doTheAction() nextId += 1 } } else if (id < nextId) sender() ! rc // otherwise an older command was lost, so wait for the resend } def receiveRecover = { case ReceiptConfirmed(id, _) => nextId = id + 1 } private def doTheAction(): Unit = () // actually do the action }
  29. 29. Exactly-Once Delivery with Akka 29 class PersistThenAct extends PersistentActor { var nextId = 1 def receiveCommand = { case FollowUpCommand(x, id, deliveryId) => val rc = ReceiptConfirmed(id, deliveryId) if (id == nextId) { persist(rc) { rc => sender() ! rc doTheAction() nextId += 1 } } else if (id < nextId) sender() ! rc // otherwise an older command was lost, so wait for the resend } def receiveRecover = { case ReceiptConfirmed(id, _) => nextId = id + 1 } private def doTheAction(): Unit = () // actually do the action }
  30. 30. Exactly-Once Delivery with Akka 30 class PersistThenAct extends PersistentActor { var nextId = 1 def receiveCommand = { case FollowUpCommand(x, id, deliveryId) => val rc = ReceiptConfirmed(id, deliveryId) if (id == nextId) { persist(rc) { rc => sender() ! rc doTheAction() nextId += 1 } } else if (id < nextId) sender() ! rc // otherwise an older command was lost, so wait for the resend } def receiveRecover = { case ReceiptConfirmed(id, _) => nextId = id + 1 } private def doTheAction(): Unit = () // actually do the action }
  31. 31. Exactly-Once Delivery with Akka 31 class PersistThenAct extends PersistentActor { var nextId = 1 def receiveCommand = { case FollowUpCommand(x, id, deliveryId) => val rc = ReceiptConfirmed(id, deliveryId) if (id == nextId) { doTheAction() persist(rc) { rc => sender() ! rc nextId += 1 } } else if (id < nextId) sender() ! rc // otherwise an older command was lost, so wait for the resend } def receiveRecover = { case ReceiptConfirmed(id, _) => nextId = id + 1 } private def doTheAction(): Unit = () // actually do the action }
  32. 32. The Hard Truth CPU can stop between any two instructions, therefore exactly-once semantics cannot be Guaranteed*. *with a capital G, i.e. with 100% absolute coverage 32
  33. 33. Mitigating that Hard Truth • probability of loss or duplication can be small • many systems are fine with almost-exactly-once • still need to decide on which side to err • persist-then-act ➟ almost-exactly-but-at-most-once • act-then-persist ➟ almost-exactly-but-at-least-once 33
  34. 34. We Can Do Better! • an operation is idempotent if multiple application produces the same result as single application • (side-)effects must also be idempotent • unique transaction identifiers etc. • this achieves effectively-exactly-once • can save persistence round-trip latency by optimistic execution 34
  35. 35. Effectively-Exactly-Once with Akka 35 class Idempotent(other: ActorRef) extends PersistentActor { var nextId = 1 def receiveCommand = { case FollowUpCommand(x, id, deliveryId) => val rc = ReceiptConfirmed(id, deliveryId) if (id == nextId) { // optimistically execute if (isValid(x)) { updateState(x) other ! Command(x) // assuming idempotent receiver } else sender() ! Invalid(x) nextId += 1 // then take care of reliable delivery handling persistAsync(rc) { rc => sender() ! rc } } else if (id < nextId) sender() ! rc } ... }
  36. 36. The Future is Bright Again! 36 source: http://freewallpapersbackgrounds.com/
  37. 37. To Recapitulate: 37 Problem Choices Transactional Storage monolithic or distributed/partitioned, serializable or eventually consistent Cluster Management strong consensus or convergent gossip, consistency or availability Message Delivery at-most-once, at-least-once, almost-exactly-once, effectively-exactly-once … …
  38. 38. Conclusion: Consistency à la Carte • not all parts of a system have the same needs • expend the effort only where necessary • idempotency does not come for free • confirmation handling cannot always be abstracted away • incur the runtime overhead only where needed • persistence round-trips take time • consensus protocols take more time • include required semantics in system design 38
  39. 39. Remember that the trade-off is always between consistency and availability. And latency.
  40. 40. ©Typesafe 2014 – All Rights Reserved

×