Download to read offline
Protect your users with Circuit breakers
- 1. Protect your userswith Circuit Breakers Scott Triglia Europython 2016 1 Jim’s
- 2. 2 Yelp’s Mission: Connecting peoplewith great local businesses.
- 3. Yelp Stats As ofQ1 2016 90M 3270%102M
- 4.
- 5. Let’s talk CircuitBreakers 5
- 6.
- 7.
- 8.
- 9.
- 10.
- 11. Our goals today: introducea basic circuit breaker 11
- 12. Our goals today: amodular circuit breaker 12
- 13. Our goals today: testit out on several scenarios 13
- 14.
- 15.
- 16.
- 17. the fundamental rule: yoursystems will fail what’s your response? 17
- 18.
- 19.
- 20.
- 21.
- 22.
- 23.
- 24.
- 25.
- 26. 26 Circuit Breaker States: *Healthy (or “closed”) * Recovering (or “half-open”) * Unhealthy (or “open”)
- 27.
- 28. 28 Recovery: * Wait forrecovery_timeout seconds * Send a trial request, trust its results
- 29. 29 Before a circuitbreaker:
- 30. 30 Before a circuitbreaker: * Diners wait forever to get food
- 31. 31 Before a circuitbreaker: * Diners wait forever to get food * Kitchen has a growing backlog
- 32. 32 Before a circuitbreaker: * Diners wait forever to get food * Kitchen has a growing backlog * New diners making things worse
- 33. 33 With a circuitbreaker:
- 34. 34 With a circuitbreaker: * Fewer frustrated users
- 35. 35 With a circuitbreaker: * Fewer frustrated users * Reduced load on the backend
- 36. 36 With a circuitbreaker: * Fewer frustrated users * Reduced load on the backend * A well defined failure mode
- 37.
- 38.
- 39.
- 40.
- 41.
- 42. 42 Should our waitersall agree? Module 1:
- 43.
- 44.
- 45. 45 New Behavior: * Clientsinform each other * Processes are no longer independent
- 46. 46 * Propagate failurefaster * Requires distributed datastore * Forces decisions about consistency
- 47. 47 What should wedo in response? Module 2:
- 48.
- 49.
- 50.
- 51.
- 52. 52 New Behavior: * Codecan check in advance about healthiness of system * Automatic monitoring!
- 53. 53 * Build featureson top of system health status * Requires a single source of truth?
- 54. 54 Who decides we’reunhealthy? Module 3:
- 55.
- 56. 56 def signal_overload(cb): if len(jobs)> THRESH: cb.mark_unhealthy()
- 57. 57 New Behavior: * CBgets signals from anywhere * Signal combining logic
- 58. 58 * Allows many(many) new signals * Must combine signals * Adds complexity to system
- 59. 59 How do werecover? Module 4:
- 60.
- 61. 61 Dark launch: * Rejectbut process normally * Dangerous with side effects Block User Request Try to process anyway!
- 62. 62 Synthetic: * Dark launchingwith fake requests * Not necessarily representative Block User Request Process fake requests
- 63. 63 New Behavior: * Trafficdetermines health * Removal of recovery timeouts
- 64. 64 * Faster(?) recovery *No timeout tuning required * Dark launching not always possible * Synthetic can be unrepresentative
- 65.
- 66. 66 Your system willfail, have a plan!
- 67. 67 The basic CBis better than nothing
- 68. 68 Questions to ask: *Should our waiters all agree? * How should I deal with unhealthiness? * Who decides we’re unhealthy? * How do we recover?
- 69. 69 Questions to ask: *Should our waiters all agree? * How should I deal with unhealthiness? * Who decides we’re unhealthy? * How do we recover?
- 70. 70 Questions to ask: *Should our waiters all agree? * How should I deal with unhealthiness? * Who decides we’re unhealthy? * How do we recover?
- 71. 71 Questions to ask: *Should our waiters all agree? * How should I deal with unhealthiness? * Who decides we’re unhealthy? * How do we recover?
- 72. 72 Questions to ask: *Should our waiters all agree? * How should I deal with unhealthiness? * Who decides we’re unhealthy? * How do we recover?
- 73.
- 74. Much comes downto your use case 74
- 75.
- 77. 77 Can’t we dobetter than rejecting requests?
- 78.
- 79. 79 How do Isafely test out a new circuit breaker?
- 80.