1. Dr. Roland Kuhn
Akka Tech Lead
@rolandkuhn
Distributed in Space and Time

2. The Future is Bright
2
source: http://nature.desktopnexus.com/wallpaper/58502/

3. The Future is Bright
• Computing Resources: Effectively Unbounded
• 18-core CPU packages, 1024 around the corner
• on-demand provisioning in the cloud
• Demand for Web-Scale Apps: Ravenous
• everyone has the tools
• successful ideas are bought for Giga-$
3

4. How do I make a Large-Scale App?
• figure out a service to provide to everyone
• this will require resources accordingly:
• storage
• computation
• communication
• scalability implies no shared contention points
• need to be elastic ➟ use the cloud
4

5. How do I make an Always-Available App?
• must not have single point of failure
• all functions and resources are replicated
• protect against data loss
• protect against service downtime
• regional content delivery
• need to distribute ➟ use the cloud
5

6. How do I make a Persistent App?
• runtime replication usually not sufficient
• system of record needed
• one central database is not tenable
• persist “locally” and replicate changes
6

7. How do I make a Persistent App?
7

8. How do I make a Persistent App?
8

9. How do I make a Persistent App?
• runtime replication usually not sufficient
• system of record needed
• one central database is not tenable
• persist “locally” and replicate changes
• temporally distributed change records
• spatially distributed change log
• granularity can be one Aggregate Root (Event Sourcing)
• extract business intelligence from history
• scale & distribute persistence ➟ in the cloud
9

10. As Noted Earlier: The Future is Bright
10
source: http://nature.desktopnexus.com/wallpaper/58502/

11. … or is it?
11
Photograph: Patrick Cromerford, 2011

12. Distribution: a Double-Edged Sword
• did that message make it over the network?
• is that other machine running?
• are our data centers in sync?
• did we lose messages in that hardware crash?
12

13. The Essence of the Problem
A distributed system is one whose parts can fail
independently of each other.
13

14. Fighting Back!
14
source: www.hear2heal.com (Kokopelli Rain Dance)

15. Transactional Storage
Consensus
Message Delivery
…

16. Transactional Storage
• Serializability simplifies reasoning about behavior
• … but is at odds with distribution (CAP theorem)
• Eventual Consistency is often sufficient
(PeterBailis:ProbabilisticallyBoundedStaleness)
• Scalability can be regained by partitioning
(PatHelland:LifebeyondDistributedTransactions)
• consistency has a cost
• you need to choose how much is necessary
16

17. PAXOS and Friends
• problem: distributed consensus
• hard solutions favor Consistency
• 2PC, PAXOS, RAFT
• if a node is unreachable then consensus cannot be reached
• soft solutions favor Availability
• quorum, allow nodes to drop out
• partitions can lead to more than one active set
• consistency always has a cost
• you need to choose which one is appropriate
17

18. Fighting Message Loss
• resending is the only cure
• sender stores the message and retries
• recipient must send acknowledgement eventually
• fighting unreliable medium is sender’s obligation
• sender’s buffer must be persistent
• resending must begin anew after a crash
18

19. At-Least-Once Delivery with Akka
19
class ALOD(other: ActorPath) extends PersistentActor with AtLeastOnceDelivery {
val IDs = Iterator from 1
def receiveCommand = {
case Command(x) =>
persist(ImportantResult(x)) { ir =>
sender() ! Done(x)
deliver(other, FollowUpCommand(x, IDs.next(), _))
}
case rc: ReceiptConfirmed =>
persist(rc) { rc => confirmDelivery(rc.deliveryId) }
}
def receiveRecover = {
case ImportantResult(x) => deliver(other, FollowUpCommand(x, IDs.next(), _))
case ReceiptConfirmed(_, id) => confirmDelivery(id)
}
}

20. At-Least-Once Delivery with Akka
20
class ALOD(other: ActorPath) extends PersistentActor with AtLeastOnceDelivery {
val IDs = Iterator from 1
def receiveCommand = {
case Command(x) =>
persist(ImportantResult(x)) { ir =>
sender() ! Done(x)
deliver(other, FollowUpCommand(x, IDs.next(), _))
}
case rc: ReceiptConfirmed =>
persist(rc) { rc => confirmDelivery(rc.deliveryId) }
}
def receiveRecover = {
case ImportantResult(x) => deliver(other, FollowUpCommand(x, IDs.next(), _))
case ReceiptConfirmed(_, id) => confirmDelivery(id)
}
}

21. At-Least-Once Delivery with Akka
21
class ALOD(other: ActorPath) extends PersistentActor with AtLeastOnceDelivery {
val IDs = Iterator from 1
def receiveCommand = {
case Command(x) =>
persist(ImportantResult(x)) { ir =>
sender() ! Done(x)
deliver(other, FollowUpCommand(x, IDs.next(), _))
}
case rc: ReceiptConfirmed =>
persist(rc) { rc => confirmDelivery(rc.deliveryId) }
}
def receiveRecover = {
case ImportantResult(x) => deliver(other, FollowUpCommand(x, IDs.next(), _))
case ReceiptConfirmed(_, id) => confirmDelivery(id)
}
}

22. At-Least-Once Delivery with Akka
22
class ALOD(other: ActorPath) extends PersistentActor with AtLeastOnceDelivery {
val IDs = Iterator from 1
def receiveCommand = {
case Command(x) =>
persist(ImportantResult(x)) { ir =>
sender() ! Done(x)
deliver(other, FollowUpCommand(x, IDs.next(), _))
}
case rc: ReceiptConfirmed =>
persist(rc) { rc => confirmDelivery(rc.deliveryId) }
}
def receiveRecover = {
case ImportantResult(x) => deliver(other, FollowUpCommand(x, IDs.next(), _))
case ReceiptConfirmed(_, id) => confirmDelivery(id)
}
}

23. At-Least-Once Delivery with Akka
23
class ALOD(other: ActorPath) extends PersistentActor with AtLeastOnceDelivery {
val IDs = Iterator from 1
def receiveCommand = {
case Command(x) =>
persist(ImportantResult(x)) { ir =>
sender() ! Done(x)
deliver(other, FollowUpCommand(x, IDs.next(), _))
}
case rc: ReceiptConfirmed =>
persist(rc) { rc => confirmDelivery(rc.deliveryId) }
}
def receiveRecover = {
case ImportantResult(x) => deliver(other, FollowUpCommand(x, IDs.next(), _))
case ReceiptConfirmed(_, id) => confirmDelivery(id)
}
}

24. Mysterious Double-Bookings
• acknowledgements can be lost
• sender will faithfully duplicate the message
• therefore named at-least-once
• recipient is responsible for dealing with this
24

25. Exactly-Once Delivery
• processing occurs only once for each message
• all middleware does it*
• EIP: Guaranteed Delivery just means persistence
• JMS: configure session for AUTO_ACKNOWLEDGE
• Google’s MillWheel
25

26. Exactly-Once Delivery with Akka
26
class PersistThenAct extends PersistentActor {
var nextId = 1
def receiveCommand = {
case FollowUpCommand(x, id, deliveryId) =>
val rc = ReceiptConfirmed(id, deliveryId)
if (id == nextId) {
persist(rc) { rc =>
sender() ! rc
doTheAction()
nextId += 1
}
} else if (id < nextId) sender() ! rc
// otherwise an older command was lost, so wait for the resend
}
def receiveRecover = {
case ReceiptConfirmed(id, _) => nextId = id + 1
}
private def doTheAction(): Unit = () // actually do the action
}

27. Exactly-Once Delivery with Akka
27
class PersistThenAct extends PersistentActor {
var nextId = 1
def receiveCommand = {
case FollowUpCommand(x, id, deliveryId) =>
val rc = ReceiptConfirmed(id, deliveryId)
if (id == nextId) {
persist(rc) { rc =>
sender() ! rc
doTheAction()
nextId += 1
}
} else if (id < nextId) sender() ! rc
// otherwise an older command was lost, so wait for the resend
}
def receiveRecover = {
case ReceiptConfirmed(id, _) => nextId = id + 1
}
private def doTheAction(): Unit = () // actually do the action
}

28. Exactly-Once Delivery with Akka
28
class PersistThenAct extends PersistentActor {
var nextId = 1
def receiveCommand = {
case FollowUpCommand(x, id, deliveryId) =>
val rc = ReceiptConfirmed(id, deliveryId)
if (id == nextId) {
persist(rc) { rc =>
sender() ! rc
doTheAction()
nextId += 1
}
} else if (id < nextId) sender() ! rc
// otherwise an older command was lost, so wait for the resend
}
def receiveRecover = {
case ReceiptConfirmed(id, _) => nextId = id + 1
}
private def doTheAction(): Unit = () // actually do the action
}

29. Exactly-Once Delivery with Akka
29
class PersistThenAct extends PersistentActor {
var nextId = 1
def receiveCommand = {
case FollowUpCommand(x, id, deliveryId) =>
val rc = ReceiptConfirmed(id, deliveryId)
if (id == nextId) {
persist(rc) { rc =>
sender() ! rc
doTheAction()
nextId += 1
}
} else if (id < nextId) sender() ! rc
// otherwise an older command was lost, so wait for the resend
}
def receiveRecover = {
case ReceiptConfirmed(id, _) => nextId = id + 1
}
private def doTheAction(): Unit = () // actually do the action
}

30. Exactly-Once Delivery with Akka
30
class PersistThenAct extends PersistentActor {
var nextId = 1
def receiveCommand = {
case FollowUpCommand(x, id, deliveryId) =>
val rc = ReceiptConfirmed(id, deliveryId)
if (id == nextId) {
persist(rc) { rc =>
sender() ! rc
doTheAction()
nextId += 1
}
} else if (id < nextId) sender() ! rc
// otherwise an older command was lost, so wait for the resend
}
def receiveRecover = {
case ReceiptConfirmed(id, _) => nextId = id + 1
}
private def doTheAction(): Unit = () // actually do the action
}

31. Exactly-Once Delivery with Akka
31
class PersistThenAct extends PersistentActor {
var nextId = 1
def receiveCommand = {
case FollowUpCommand(x, id, deliveryId) =>
val rc = ReceiptConfirmed(id, deliveryId)
if (id == nextId) {
doTheAction()
persist(rc) { rc =>
sender() ! rc
nextId += 1
}
} else if (id < nextId) sender() ! rc
// otherwise an older command was lost, so wait for the resend
}
def receiveRecover = {
case ReceiptConfirmed(id, _) => nextId = id + 1
}
private def doTheAction(): Unit = () // actually do the action
}

32. The Hard Truth
CPU can stop between any two instructions, therefore
exactly-once semantics cannot be Guaranteed*.
*with a capital G, i.e. with 100% absolute coverage
32

33. Mitigating that Hard Truth
• probability of loss or duplication can be small
• many systems are fine with almost-exactly-once
• still need to decide on which side to err
• persist-then-act ➟ almost-exactly-but-at-most-once
• act-then-persist ➟ almost-exactly-but-at-least-once
33

34. We Can Do Better!
• an operation is idempotent if multiple application
produces the same result as single application
• (side-)effects must also be idempotent
• unique transaction identifiers etc.
• this achieves effectively-exactly-once
• can save persistence round-trip latency by
optimistic execution
34

35. Effectively-Exactly-Once with Akka
35
class Idempotent(other: ActorRef) extends PersistentActor {
var nextId = 1
def receiveCommand = {
case FollowUpCommand(x, id, deliveryId) =>
val rc = ReceiptConfirmed(id, deliveryId)
if (id == nextId) {
// optimistically execute
if (isValid(x)) {
updateState(x)
other ! Command(x) // assuming idempotent receiver
} else sender() ! Invalid(x)
nextId += 1
// then take care of reliable delivery handling
persistAsync(rc) { rc =>
sender() ! rc
}
} else if (id < nextId) sender() ! rc
}
...
}

36. The Future is Bright Again!
36
source: http://freewallpapersbackgrounds.com/

37. To Recapitulate:
37
Problem Choices
Transactional Storage
monolithic or distributed/partitioned,
serializable or eventually consistent
Cluster Management
strong consensus or convergent gossip,
consistency or availability
Message Delivery
at-most-once, at-least-once,
almost-exactly-once, effectively-exactly-once
… …

38. Conclusion: Consistency à la Carte
• not all parts of a system have the same needs
• expend the effort only where necessary
• idempotency does not come for free
• confirmation handling cannot always be abstracted away
• incur the runtime overhead only where needed
• persistence round-trips take time
• consensus protocols take more time
• include required semantics in system design
38

39. Remember that the trade-off is always between
consistency and availability.
And latency.

40. ©Typesafe 2014 – All Rights Reserved