Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Without Resilience
Nothing Else Matters
Jonas Bonér
CTO Lightbend
@jboner
This Is Fault Tolerance
“But it ain’t how hard you’re hit;
it’s about how hard you can get
hit, and keep moving forward.
H...
Resilience
Is Beyond
Fault
Tolerance
Resilience
“The ability of a substance or
object to spring back into shape.
The capacity to recover quickly
from difficult...
Antifragility
“Antifragility is beyond resilience and
robustness. The resilient resists shock and
stays the same; the anti...
“We can model and understand in isolation. 

But, when released into competitive nominally
regulated societies, their conn...
Software Systems Today Are
Incredibly Complex
Netflix Twitter
We need to study
Resilience in
Complex
Systems
Complicated System
Complex System
Complicated ≠Complex
“Complex systems run in degraded mode.”
“Complex systems run as broken systems.”
- richard Cook
How Complex Systems Fail -...
“Counterintuitive. That’s [Jay] Forrester’s
word to describe complex systems.
Leverage points are not intuitive. Or if the...
“Humans should not be
involved in setting timeouts.”
“Human involvement in
complex systems is the biggest
source of troubl...
Humans Generally
Make Things Worse
‘‘Going solid’’: a model of system dynamics and consequences for patient safety - R Cook, J Rasmussen
Resilience in comple...
Economic
Failure
Boundary
Unacceptable
Workload
Boundary
Accident
Boundary
Management Pressure
Towards Economic Efficiency...
‘‘Going solid’’: a model of system dynamics and consequences for patient safety - R Cook, J Rasmussen
Resilience in comple...
‘‘Going solid’’: a model of system dynamics and consequences for patient safety - R Cook, J Rasmussen
Resilience in comple...
‘‘Going solid’’: a model of system dynamics and consequences for patient safety - R Cook, J Rasmussen
Resilience in comple...
Embrace
Failure
Resilience
is by
Design
Photo courtesy of FEMA/Joselyne Augustino
“Autonomy makes information local,
leading to greater certainty and stability.”
- Mark Burgess
In Search of Certainty - Ma...
Promise Theory
Think in
Promises
Not
Commands
Promise Theory
Promises converge towards
A definite outcome from
unpredictable beginnings
improved Stability
Commands dive...
Resilience in
Biological
Systems
MeerkatsPuppies! Now that I’ve got your attention, complexity theory - Nicolas Perony, TED talk
“In three words, in the animal kingdom,
simplicity leads to complexity 

which leads to resilience.”
- Nicolas Perony
Pupp...
Resilience in
Social
Systems
Dealing in Security
Understanding vital services, and how they keep you safe
1INDIVIDUAL (you)
6ways to die
3sets of essen...
What we can learn from
Resilience in
Biological and
Social Systems
1. Feature Diversity and redundancy
2. Inter-Connected ...
Resilience in
Computer
Systems
We Need To
Manage
Failure
Not Try To Avoid It
Let It
Crash
Crash
Only
Software
Crash-Only Software - George Candea, Armando Fox
Stop = Crash Safely
Start = Recover Fast
Recursive Restartability
Turning the Crash-Only Sledgehammer into a Scalpel
Recursive Restartability: Turning the Reboot S...
Traditional
State Management
Object
Critical state
that needs protection
Client
Thread boundary
Synchronous dispatch Threa...
“Accidents come from relationships 
not broken parts.”
- Sidney dekker
Drift into Failure - Sidney Dekker
Requirements for a
Sane Failure Model
1. Contained—Avoid cascading failures
2. Reified—as messages
3. Signalled—Asynchrono...
Bulkhead
Pattern
Enter Supervision
Out of the Tar Pit - Ben Moseley , Peter Marks
• Input Data
• Derived Data
Critical
We need a way out of the
State Tar Pit
Essential
State
Out of the Tar Pit - Ben Moseley , Peter Marks
Essential
Logic
Accidental
State and
Control
We need a way ...
The
Vending
Machine
Pattern
Think Vending Machine
Coffee
Machine
Programmer
Inserts coins
Gets coffee
Add more coins
Think Vending Machine
Programmer
Service
Guy
Inserts coins
Gets coffee
Out of
coffee beans
failure
Adds
more
beans
Out of ...
Think Vending Machine
ServiceClient
Supervisor
Request
Response
Validation Error
Application
Failure
Manages
Failure
Error
Kernel
Pattern
Onion-layered state & Failure management
Making reliable distributed systems in the presence of softw...
Onion Layered
State Management
Error Kernel
Object
Critical state
that needs protection
Client
Supervision
Supervision
Thr...
Demo
Time
Let’s model a resilient vending machine, in Akka
Demo Runner
object VendingMachineDemo extends App {



val system = ActorSystem("vendingMachineDemo")

val coffeeMachine =...
Test Happy Path
// Insert 2 coins and get an Espresso

customer.send(coffeeMachine, Coins(2))

customer.send(coffeeMachine...
Test User Error
customer.send(coffeeMachine, Coins(1))

customer.send(coffeeMachine, Selection(Latte))

val NotEnoughCoins...
Test System Failure
// Insert 1 coin (had 1 before) and try to get my Latte

// Machine should:

// 1. Fail

// 2. Restart...
Protocol
// Coffee types

trait CoffeeType

case object BlackCoffee extends CoffeeType

case object Latte extends CoffeeTy...
CoffeeMachine
class CoffeeMachine extends Actor {

val price = 2

var nrOfInsertedCoins = 0

var outOfCoffeeBeans = false
...
CoffeeMachine
def receive = {

case Coins(nr) =>

nrOfInsertedCoins += nr

totalNrOfCoins += nr

println(s"Inserted [$nr] ...
CoffeeMachine
override def postRestart(failure: Throwable): Unit = {

println(s"Restarting coffee machine...")

failure ma...
Supervisor
class CoffeeMachineManager extends Actor {

override val supervisorStrategy =

OneForOneStrategy(maxNrOfRetries...
So.........
Sorry...but Not really.
Are We Done?
We can not
keep putting
all eggs in the
same basket
We need to
Maintain Diversity
and Redundancy
The Network
is Reliable
NOT
Really
Here, We are living in the
Looming
Shadow of
Impossibility
Theorems
CAP: Consistency is impossible
FLP: Consensus is impos...
Towards Resilient Distributed Systems
Isolation
• Autonomous Microservices
• Resilient Protocols
• Virtualization
Data Res...
Microservices
1. Autonomy
2. Isolation
3. Mobility
4. Single Responsibility
5. Exclusive StatE
An autonomous Service
can only promise
its own behavior
Apply Promise Theory
We need to decompose the system using
Consistency Boundaries
Inside Data
Our current present—state
Outside Data
Blast from the past—facts
Between Services
Hope for the future—commands...
WITHIN the Consistency Boundary
we can have STRONG CONSISTENCY
BETWEEN
Consistency
Boundaries
it is a
ZOO
We need Systems that are Decoupled in
Time and Space
Embrace the Network
•Go Asynchronous
•Make distribution first class
•Learn from the mistakes of RPC, EJB & CORBA
•Leverage...
Location Transparency
One communication abstraction
across all dimensions of scale
Core Socket CPU
Container Server Rack
D...
Resilient Protocols
are tolerant to
•Message loss
•Message reordering
•Message duplication
Embrace ACID 2.0
• Associative
...
“To make a system of interconnected components
crash-only, it must be designed so that components
can tolerate the crashes...
"Software components should be designed such
that they can deny service for any request or call.
Then, if an underlying co...
Services need to learn to accept
NO for an answer
Member
Node
Member
Node
Member
Node
Member
Node
Member
Node
Member
Node
Member
Node
Member
Node
Member
Node
Member
Node
De...
STRONG
Consistency
Is the wrong default
“Two-phase commit is the
anti-availability protocol.”
- Pat Helland
Standing on Distributed Shoulders of Giants - Pat Hell...
Eventual
Consistency
We have to rely on
But relax, it’s how the world works
Transactions
But I really need
“In general, application developers
simply do not implement large
scalable applications assuming
distributed transactions....
Guess.
Apologize.
Compensate.
Use a protocol of
“The truth is the log. The database is a
cache of a subset of the log.”
- Pat Helland
Immutability Changes Everything - Pa...
CRUD is DEAD
Event Logging
• Work with Facts—immutable values
• Event Sourcing
• DB of Facts—Keep all history
• Just replay on failure
...
Let’s model a resilient & Event Logged vending machine, in Akka
Demo
Time
Event Logged CoffeeMachine
// Events

case class CoinsReceived(number: Int)

class CoffeeMachine extends PersistentActor {...
“An escalator can never break: it can only
become stairs. You should never see an
Escalator Temporarily Out Of Order sign,...
Graceful
Degradation
Circuit Breaker
Little’s Law
L = λW
Queue Length = Arrival Rate * Response Time
W = L/λ
Response Time = Queue Length / Arrival Rate
W: Res...
Flow Control
Always Apply BackPressure
Feedback
Control
“Continuously compare the actual
output to its desired reference value;
then apply a change to the system
inputs that coun...
Feedback Control
Influencing a
Complex System
Places to Intervene
in a Complex System
1. The constants, parameters or numbers
2. The sizes of buffers relative to their ...
Triple Loop Learning
Loop 1: Follow the rules
Loop 2: Change the rules
Loop 3: Learn how to learn
Triple Loop Learning - C...
Testing
What can we learn from Arnold?
Blow things up
Shoot
Your App
Down
Pull the Plug
…and see what happens
Executive
Summary
“Complex systems run as broken systems.”
- richard Cook
How Complex Systems Fail - Richard Cook
Resilience
is by
Design
Photo courtesy of FEMA/Joselyne Augustino
Without Resilience
Nothing Else Matters
References
Drift into Failure - http://www.amazon.com/Drift-into-Failure-Components-Understanding-ebook/dp/B009KOKXKY

How...
Thank
You
Without Resilience
Nothing Else Matters
Jonas Bonér
CTO Lightbend
@jboner
Without Resilience, Nothing Else Matters
Without Resilience, Nothing Else Matters
Without Resilience, Nothing Else Matters
Upcoming SlideShare
Loading in …5
×

Without Resilience, Nothing Else Matters

13,442 views

Published on

It doesn’t matter how beautiful, loosely coupled, scalable, highly concurrent, non-blocking, responsive and performant your application is—if it isn't running, then it's 100% useless. Without resilience, nothing else matters.

Most developers understand what the word resilience means, at least superficially, but way too many lack a deeper understanding of what it really means in the context of the system that they are working on now. I find it really sad to see, since understanding and managing failure is more important today than ever. Outages are incredibly costly—for many definitions of cost—and can sometimes take down whole businesses.

In this talk we will explore the essence of resilience. What does it really mean? What is its mechanics and characterizing traits? How do other sciences and industries manage it, and what can we learn from that? We will see that everything hints at the same conclusion; that failure is inevitable and needs to be embraced, and that resilience is by design.

Published in: Engineering
  • Hello! High Quality And Affordable Essays For You. Starting at $4.99 per page - Check our website! https://vk.cc/82gJD2
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

Without Resilience, Nothing Else Matters

  1. 1. Without Resilience Nothing Else Matters Jonas Bonér CTO Lightbend @jboner
  2. 2. This Is Fault Tolerance “But it ain’t how hard you’re hit; it’s about how hard you can get hit, and keep moving forward. How much you can take, and keep moving forward. That’s how winning is done.” - Rocky Balboa
  3. 3. Resilience Is Beyond Fault Tolerance
  4. 4. Resilience “The ability of a substance or object to spring back into shape. The capacity to recover quickly from difficulties.” -Merriam Webster
  5. 5. Antifragility “Antifragility is beyond resilience and robustness. The resilient resists shock and stays the same; the antifragile gets better.” - Nassem Nicholas Taleb Antifragile: Things That Gain from Disorder - Nassim Nicholas Taleb
  6. 6. “We can model and understand in isolation. 
 But, when released into competitive nominally regulated societies, their connections proliferate, 
 their interactions and interdependencies multiply, 
 their complexities mushroom. 
 And we are caught short.” - Sidney Dekker Drift into Failure - Sidney Dekker
  7. 7. Software Systems Today Are Incredibly Complex Netflix Twitter
  8. 8. We need to study Resilience in Complex Systems
  9. 9. Complicated System
  10. 10. Complex System
  11. 11. Complicated ≠Complex
  12. 12. “Complex systems run in degraded mode.” “Complex systems run as broken systems.” - richard Cook How Complex Systems Fail - Richard Cook
  13. 13. “Counterintuitive. That’s [Jay] Forrester’s word to describe complex systems. Leverage points are not intuitive. Or if they are, we intuitively use them backward, systematically worsening whatever problems we are trying to solve.” - Donella Meadows Leverage Points: Places to Intervene in a System - Donella Meadows
  14. 14. “Humans should not be involved in setting timeouts.” “Human involvement in complex systems is the biggest source of trouble.” - Ben Christensen, Netflix
  15. 15. Humans Generally Make Things Worse
  16. 16. ‘‘Going solid’’: a model of system dynamics and consequences for patient safety - R Cook, J Rasmussen Resilience in complex adaptive systems: Operating at the Edge of Failure - Richard Cook - Talk at Velocity NY 2013 Economic Failure Boundary Unacceptable Workload Boundary Operating Point FAILURE Accident Boundary Operating at the Edge of Failure
  17. 17. Economic Failure Boundary Unacceptable Workload Boundary Accident Boundary Management Pressure Towards Economic Efficiency Gradient Towards Least Effort Counter Gradient For More Resilience ‘‘Going solid’’: a model of system dynamics and consequences for patient safety - R Cook, J Rasmussen Resilience in complex adaptive systems: Operating at the Edge of Failure - Richard Cook - Talk at Velocity NY 2013 Operating at the Edge of Failure
  18. 18. ‘‘Going solid’’: a model of system dynamics and consequences for patient safety - R Cook, J Rasmussen Resilience in complex adaptive systems: Operating at the Edge of Failure - Richard Cook - Talk at Velocity NY 2013 Economic Failure Boundary Unacceptable Workload Boundary Accident Boundary Error Margin Marginal Boundary Operating at the Edge of Failure
  19. 19. ‘‘Going solid’’: a model of system dynamics and consequences for patient safety - R Cook, J Rasmussen Resilience in complex adaptive systems: Operating at the Edge of Failure - Richard Cook - Talk at Velocity NY 2013 Accident Boundary Marginal Boundary ? Operating at the Edge of Failure
  20. 20. ‘‘Going solid’’: a model of system dynamics and consequences for patient safety - R Cook, J Rasmussen Resilience in complex adaptive systems: Operating at the Edge of Failure - Richard Cook - Talk at Velocity NY 2013 Operating at the Edge of Failure Accident Boundary Marginal Boundary
  21. 21. Embrace Failure
  22. 22. Resilience is by Design Photo courtesy of FEMA/Joselyne Augustino
  23. 23. “Autonomy makes information local, leading to greater certainty and stability.” - Mark Burgess In Search of Certainty - Mark Burgess
  24. 24. Promise Theory Think in Promises Not Commands
  25. 25. Promise Theory Promises converge towards A definite outcome from unpredictable beginnings improved Stability Commands diverge into unpredictable outcomes from definite beginnings decreased Stability
  26. 26. Resilience in Biological Systems
  27. 27. MeerkatsPuppies! Now that I’ve got your attention, complexity theory - Nicolas Perony, TED talk
  28. 28. “In three words, in the animal kingdom, simplicity leads to complexity 
 which leads to resilience.” - Nicolas Perony Puppies! Now that I’ve got your attention, complexity theory - Nicolas Perony, TED talk
  29. 29. Resilience in Social Systems
  30. 30. Dealing in Security Understanding vital services, and how they keep you safe 1INDIVIDUAL (you) 6ways to die 3sets of essential services 7layers of PROTECTION Dealing in Security - Mike Bennet, Vinay Gupta
  31. 31. What we can learn from Resilience in Biological and Social Systems 1. Feature Diversity and redundancy 2. Inter-Connected network structure 3. Wide distribution across all scales 4. Capacity to self-adapt & self-organize Toward Resilient Architectures 1: Biology Lessons - Michael Mehaffy, Nikos A. Salingaros Applying resilience thinking: Seven principles for building resilience in social-ecological systems - Reinette Biggs et. al.
  32. 32. Resilience in Computer Systems
  33. 33. We Need To Manage Failure Not Try To Avoid It
  34. 34. Let It Crash
  35. 35. Crash Only Software Crash-Only Software - George Candea, Armando Fox Stop = Crash Safely Start = Recover Fast
  36. 36. Recursive Restartability Turning the Crash-Only Sledgehammer into a Scalpel Recursive Restartability: Turning the Reboot Sledgehammer into a Scalpel - George Candea, Armando Fox
  37. 37. Traditional State Management Object Critical state that needs protection Client Thread boundary Synchronous dispatch Thread boundary ? Utterly broken
  38. 38. “Accidents come from relationships  not broken parts.” - Sidney dekker Drift into Failure - Sidney Dekker
  39. 39. Requirements for a Sane Failure Model 1. Contained—Avoid cascading failures 2. Reified—as messages 3. Signalled—Asynchronously 4. Observed—by 1-N 5. Managed—Outside failed Context Failures need to be
  40. 40. Bulkhead Pattern
  41. 41. Enter Supervision
  42. 42. Out of the Tar Pit - Ben Moseley , Peter Marks • Input Data • Derived Data Critical We need a way out of the State Tar Pit
  43. 43. Essential State Out of the Tar Pit - Ben Moseley , Peter Marks Essential Logic Accidental State and Control We need a way out of the State Tar Pit
  44. 44. The Vending Machine Pattern
  45. 45. Think Vending Machine Coffee Machine Programmer Inserts coins Gets coffee Add more coins
  46. 46. Think Vending Machine Programmer Service Guy Inserts coins Gets coffee Out of coffee beans failure Adds more beans Out of coffee beans error WRONG Coffee Machine
  47. 47. Think Vending Machine ServiceClient Supervisor Request Response Validation Error Application Failure Manages Failure
  48. 48. Error Kernel Pattern Onion-layered state & Failure management Making reliable distributed systems in the presence of software errors - Joe Armstrong On Erlang, State and Crashes - Jesper Louis Andersen
  49. 49. Onion Layered State Management Error Kernel Object Critical state that needs protection Client Supervision Supervision Thread boundary Supervision
  50. 50. Demo Time Let’s model a resilient vending machine, in Akka
  51. 51. Demo Runner object VendingMachineDemo extends App {
 
 val system = ActorSystem("vendingMachineDemo")
 val coffeeMachine = system.actorOf(Props[CoffeeMachineManager], "coffeeMachineManager")
 val customer = Inbox.create(system) // emulates the customer
 … // test runs 
 system.shutdown()
 } https://gist.github.com/jboner/d24c0eb91417a5ec10a6
  52. 52. Test Happy Path // Insert 2 coins and get an Espresso
 customer.send(coffeeMachine, Coins(2))
 customer.send(coffeeMachine, Selection(Espresso))
 val Beverage(coffee1) = customer.receive(5.seconds)
 println(s"Got myself an $coffee1")
 assert(coffee1 == Espresso) https://gist.github.com/jboner/d24c0eb91417a5ec10a6
  53. 53. Test User Error customer.send(coffeeMachine, Coins(1))
 customer.send(coffeeMachine, Selection(Latte))
 val NotEnoughCoinsError(message) = customer.receive(5.seconds)
 println(s"Got myself a validation error: $message")
 assert(message == "Please insert [1] coins") https://gist.github.com/jboner/d24c0eb91417a5ec10a6
  54. 54. Test System Failure // Insert 1 coin (had 1 before) and try to get my Latte
 // Machine should:
 // 1. Fail
 // 2. Restart
 // 3. Resubmit my order
 // 4. Give me my coffee
 customer.send(coffeeMachine, Coins(1))
 customer.send(coffeeMachine, TriggerOutOfCoffeeBeansFailure)
 customer.send(coffeeMachine, Selection(Latte))
 val Beverage(coffee2) = customer.receive(5.seconds)
 println(s"Got myself a $coffee2")
 assert(coffee2 == Latte)
 https://gist.github.com/jboner/d24c0eb91417a5ec10a6
  55. 55. Protocol // Coffee types
 trait CoffeeType
 case object BlackCoffee extends CoffeeType
 case object Latte extends CoffeeType
 case object Espresso extends CoffeeType
 
 // Commands
 case class Coins(number: Int)
 case class Selection(coffee: CoffeeType)
 case object TriggerOutOfCoffeeBeansFailure
 
 // Events
 case class CoinsReceived(number: Int)
 
 // Replies
 case class Beverage(coffee: CoffeeType)
 
 // Errors
 case class NotEnoughCoinsError(message: String)
 
 // Failures
 case class OutOfCoffeeBeansFailure(customer: ActorRef,
 pendingOrder: Selection,
 nrOfInsertedCoins: Int) extends Exception https://gist.github.com/jboner/d24c0eb91417a5ec10a6
  56. 56. CoffeeMachine class CoffeeMachine extends Actor {
 val price = 2
 var nrOfInsertedCoins = 0
 var outOfCoffeeBeans = false
 var totalNrOfCoins = 0
 
 def receive = { … }
 
 override def postRestart(failure: Throwable): Unit = { … } } https://gist.github.com/jboner/d24c0eb91417a5ec10a6
  57. 57. CoffeeMachine def receive = {
 case Coins(nr) =>
 nrOfInsertedCoins += nr
 totalNrOfCoins += nr
 println(s"Inserted [$nr] coins")
 println(s"Total number of coins in machine is [$totalNrOfCoins]")
 
 case selection @ Selection(coffeeType) =>
 if (nrOfInsertedCoins < price)
 sender.tell(NotEnoughCoinsError( s”Insert [${price - nrOfInsertedCoins}] coins"), self)
 else {
 if (outOfCoffeeBeans)
 throw new OutOfCoffeeBeansFailure(sender, selection, nrOfInsertedCoins)
 println(s"Brewing your $coffeeType")
 sender.tell(Beverage(coffeeType), self)
 nrOfInsertedCoins = 0
 }
 
 case TriggerOutOfCoffeeBeansFailure =>
 outOfCoffeeBeans = true
 } https://gist.github.com/jboner/d24c0eb91417a5ec10a6
  58. 58. CoffeeMachine override def postRestart(failure: Throwable): Unit = {
 println(s"Restarting coffee machine...")
 failure match {
 case OutOfCoffeeBeansFailure(customer, pendingOrder, coins) =>
 nrOfInsertedCoins = coins
 outOfCoffeeBeans = false
 println(s"Resubmitting pending order $pendingOrder")
 context.self.tell(pendingOrder, customer)
 }
 } https://gist.github.com/jboner/d24c0eb91417a5ec10a6
  59. 59. Supervisor class CoffeeMachineManager extends Actor {
 override val supervisorStrategy =
 OneForOneStrategy(maxNrOfRetries = 10, withinTimeRange = 1.minute) {
 case e: OutOfCoffeeBeansFailure =>
 println(s"ServiceGuy notified: $e")
 Restart
 case _: Exception =>
 Escalate
 }
 
 // to simplify things he is only managing 1 single machine
 val machine = context.actorOf( Props[CoffeeMachine], name = "coffeeMachine")
 
 def receive = {
 case request => machine.forward(request)
 }
 } https://gist.github.com/jboner/d24c0eb91417a5ec10a6
  60. 60. So......... Sorry...but Not really. Are We Done?
  61. 61. We can not keep putting all eggs in the same basket
  62. 62. We need to Maintain Diversity and Redundancy
  63. 63. The Network is Reliable NOT Really
  64. 64. Here, We are living in the Looming Shadow of Impossibility Theorems CAP: Consistency is impossible FLP: Consensus is impossible
  65. 65. Towards Resilient Distributed Systems Isolation • Autonomous Microservices • Resilient Protocols • Virtualization Data Resilience • Eventual & Causal Consistency • Event Logging • Flow Control / Feedback Control Self-healing • Decentralized Architectures • Gossip Protocols • Failure Detection Embrace the Network •Asynchronicity •Location Transparency
  66. 66. Microservices 1. Autonomy 2. Isolation 3. Mobility 4. Single Responsibility 5. Exclusive StatE
  67. 67. An autonomous Service can only promise its own behavior Apply Promise Theory
  68. 68. We need to decompose the system using Consistency Boundaries
  69. 69. Inside Data Our current present—state Outside Data Blast from the past—facts Between Services Hope for the future—commands Data on the inside vs Data on the outside - Pat Helland
  70. 70. WITHIN the Consistency Boundary we can have STRONG CONSISTENCY
  71. 71. BETWEEN Consistency Boundaries it is a ZOO
  72. 72. We need Systems that are Decoupled in Time and Space
  73. 73. Embrace the Network •Go Asynchronous •Make distribution first class •Learn from the mistakes of RPC, EJB & CORBA •Leverage Location Transparency •Actor Model does it right
  74. 74. Location Transparency One communication abstraction across all dimensions of scale Core Socket CPU Container Server Rack Data Center GLobal
  75. 75. Resilient Protocols are tolerant to •Message loss •Message reordering •Message duplication Embrace ACID 2.0 • Associative • Commutative • Idempotent • Distributed Depend on • Asynchronous Communication • Eventual Consistency
  76. 76. “To make a system of interconnected components crash-only, it must be designed so that components can tolerate the crashes and temporary unavailability of their peers. This means we require: [1] strong modularity with relatively impermeable component boundaries, [2] timeout-based communication and lease-based resource allocation, and [3] self- describing requests that carry a time-to-live and information on whether they are idempotent.” - George Candea, Armando Fox Crash-Only Software - George Candea, Armando Fox
  77. 77. "Software components should be designed such that they can deny service for any request or call. Then, if an underlying component can say No, apps must be designed to take No for an answer and decide how to proceed: give up, wait and retry, reduce fidelity, etc.” - George Candea, Armando Fox Recursive Restartability: Turning the Reboot Sledgehammer into a Scalpel - George Candea, Armando Fox
  78. 78. Services need to learn to accept NO for an answer
  79. 79. Member Node Member Node Member Node Member Node Member Node Member Node Member Node Member Node Member Node Member Node Decentralized Epidemic Gossip Protocols Gossip Of membership, Data & Meta DataFailure detection heartbeat
  80. 80. STRONG Consistency Is the wrong default
  81. 81. “Two-phase commit is the anti-availability protocol.” - Pat Helland Standing on Distributed Shoulders of Giants - Pat Helland
  82. 82. Eventual Consistency We have to rely on But relax, it’s how the world works
  83. 83. Transactions But I really need
  84. 84. “In general, application developers simply do not implement large scalable applications assuming distributed transactions.” - Pat Helland Life Beyond Distributed Transactions - Pat Helland
  85. 85. Guess. Apologize. Compensate. Use a protocol of
  86. 86. “The truth is the log. The database is a cache of a subset of the log.” - Pat Helland Immutability Changes Everything - Pat Helland
  87. 87. CRUD is DEAD
  88. 88. Event Logging • Work with Facts—immutable values • Event Sourcing • DB of Facts—Keep all history • Just replay on failure • Free Auditing, Debugging, Replication • Single Writer PRinciple • Avoids OO-Relational impedence mismatch • CQRS—Separate the Read & Write Model
  89. 89. Let’s model a resilient & Event Logged vending machine, in Akka Demo Time
  90. 90. Event Logged CoffeeMachine // Events
 case class CoinsReceived(number: Int)
 class CoffeeMachine extends PersistentActor {
 val price = 2
 var nrOfInsertedCoins = 0
 var outOfCoffeeBeans = false
 var totalNrOfCoins = 0
 
 override def persistenceId = "CoffeeMachine"
 
 override def receiveCommand: Receive = {
 case Coins(nr) =>
 nrOfInsertedCoins += nr
 println(s"Inserted [$nr] coins")
 persist(CoinsReceived(nr)) { evt =>
 totalNrOfCoins += nr
 println(s"Total number of coins in machine is [$totalNrOfCoins]")
 } … } override def receiveRecover: Receive = {
 case CoinsReceived(coins) =>
 totalNrOfCoins += coins
 println(s"Total number of coins in machine is [$totalNrOfCoins]")
 }
 } https://gist.github.com/jboner/1db37eeee3ed3c9422e4
  91. 91. “An escalator can never break: it can only become stairs. You should never see an Escalator Temporarily Out Of Order sign, just Escalator Temporarily Stairs. Sorry for the convenience.” - Mitch Hedberg
  92. 92. Graceful Degradation
  93. 93. Circuit Breaker
  94. 94. Little’s Law L = λW Queue Length = Arrival Rate * Response Time W = L/λ Response Time = Queue Length / Arrival Rate W: Response Time L: Queue Length
  95. 95. Flow Control Always Apply BackPressure
  96. 96. Feedback Control
  97. 97. “Continuously compare the actual output to its desired reference value; then apply a change to the system inputs that counteracts any deviation of the actual output from the reference.” - Philipp K. Janert Feedback Control for Computer Systems - Philipp K. Janet The Feedback Principle
  98. 98. Feedback Control
  99. 99. Influencing a Complex System
  100. 100. Places to Intervene in a Complex System 1. The constants, parameters or numbers 2. The sizes of buffers relative to their flows 3. The structure of material stocks and flows 4. The lengths of delays, relative to the rate of system change 5. The strength of negative feedback loops 6. The gain around driving positive feedback loops 7. The structure of information flows 8. The rules of the system 9. The power to add, change, evolve, or self-organize structure 10. The goals of the system 11. The mindset or paradigm out of which the system arises 12. The power to transcend paradigms Leverage Points: Places to Intervene in a System - Donella Meadows:
  101. 101. Triple Loop Learning Loop 1: Follow the rules Loop 2: Change the rules Loop 3: Learn how to learn Triple Loop Learning - Chris Argyris
  102. 102. Testing
  103. 103. What can we learn from Arnold? Blow things up
  104. 104. Shoot Your App Down
  105. 105. Pull the Plug …and see what happens
  106. 106. Executive Summary
  107. 107. “Complex systems run as broken systems.” - richard Cook How Complex Systems Fail - Richard Cook
  108. 108. Resilience is by Design Photo courtesy of FEMA/Joselyne Augustino
  109. 109. Without Resilience Nothing Else Matters
  110. 110. References Drift into Failure - http://www.amazon.com/Drift-into-Failure-Components-Understanding-ebook/dp/B009KOKXKY How Complex Systems Fail - http://web.mit.edu/2.75/resources/random/How%20Complex%20Systems%20Fail.pdf Leverage Points: Places to Intervene in a System - http://www.donellameadows.org/archives/leverage-points-places-to-intervene-in-a-system/ Going Solid: A Model of System Dynamics and Consequences for Patient Safety - http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1743994/ Resilience in Complex Adaptive Systems: Operating at the Edge of Failure - https://www.youtube.com/watch?v=PGLYEDpNu60 Puppies! Now that I’ve got your attention, Complexity Theory - https://www.ted.com/talks/ nicolas_perony_puppies_now_that_i_ve_got_your_attention_complexity_theory How Bacteria Becomes Resistant - http://www.abc.net.au/science/slab/antibiotics/resistance.htm Towards Resilient Architectures: Biology Lessons - http://www.metropolismag.com/Point-of-View/March-2013/Toward-Resilient-Architectures-1-Biology-Lessons/ Dealing in Security - http://resiliencemaps.org/files/Dealing_in_Security.July2010.en.pdf What is resilience? An introduction to social-ecological research - http://www.stockholmresilience.org/download/18.10119fc11455d3c557d6d21/1398172490555/ SU_SRC_whatisresilience_sidaApril2014.pdf Applying resilience thinking: Seven principles for building resilience in social-ecological systems - http://www.stockholmresilience.org/download/ 18.10119fc11455d3c557d6928/1398150799790/SRC+Applying+Resilience+final.pdf Crash-Only Software - https://www.usenix.org/legacy/events/hotos03/tech/full_papers/candea/candea.pdf Recursive Restartability: Turning the Reboot Sledgehammer into a Scalpel - http://roc.cs.berkeley.edu/papers/recursive_restartability.pdf Out of the Tar Pit - http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.93.8928 Bulkhead Pattern - http://skife.org/architecture/fault-tolerance/2009/12/31/bulkheads.html Making Reliable Distributed Systems in the Presence of Software Errors - http://www.erlang.org/download/armstrong_thesis_2003.pdf On Erlang, State and Crashes - http://jlouisramblings.blogspot.be/2010/11/on-erlang-state-and-crashes.html Akka Supervision - http://doc.akka.io/docs/akka/snapshot/general/supervision.html Release It!: Design and Deploy Production-Ready Software - https://pragprog.com/book/mnee/release-it Feedback Control for Computer Systems - http://www.amazon.com/Feedback-Control-Computer-Systems-Philipp/dp/1449361692 The Network in Reliable - http://queue.acm.org/detail.cfm?id=2655736 Data on the Outside vs Data on the Inside - https://msdn.microsoft.com/en-us/library/ms954587.aspx Life Beyond Distributed Transactions - http://adrianmarriott.net/logosroot/papers/LifeBeyondTxns.pdf Immutability Changes Everything - http://cidrdb.org/cidr2015/Papers/CIDR15_Paper16.pdf Standing on Distributed Shoulders of Giants - https://queue.acm.org/detail.cfm?id=2953944 Thinking in Promises - http://shop.oreilly.com/product/0636920036289.do In Search Of Certainty - http://shop.oreilly.com/product/0636920038542.do Reactive Microservices Architecture - http://www.oreilly.com/programming/free/reactive-microservices-architecture-orm.csp Reactive Streams - http://reactive-streams.org Vending Machine Akka Supervision Demo - https://gist.github.com/jboner/d24c0eb91417a5ec10a6 Persistent Vending Machine Akka Supervision Demo - https://gist.github.com/jboner/1db37eeee3ed3c9422e4
  111. 111. Thank You
  112. 112. Without Resilience Nothing Else Matters Jonas Bonér CTO Lightbend @jboner

×