The document discusses circuit breakers and how they can be used to protect systems from failures. It introduces the basic concept of a circuit breaker and its states. It then explores different modules and ways circuit breakers could be enhanced, such as having processes coordinate, automatic monitoring of system health, different signals of unhealthiness, and approaches to recovery. Key questions are raised about how circuit breakers could be implemented including whether processes should coordinate, who determines unhealthiness, and how to safely test new circuit breakers.

1. Protect your users with
Circuit Breakers
Scott Triglia
VanPy Day 2016
1
Jim’s

2. 2
Yelp’s Mission:
Connecting people with great
local businesses.

3. Yelp Stats
As of Q1 2016
90M 3270%102M

4. Scott Triglia
4
@scott_triglia
Work with the $$$

5. Let’s talk Circuit Breakers
5

6. 6

7. 7

8. 8

9. 9

10. 10

11. Our goals today:
introduce a basic circuit breaker
11

12. Our goals today:
a modular circuit breaker
12

13. Our goals today:
test it out on several scenarios
13

14. 14

15. 15

16. 1616
1
2
3
4

17. the fundamental rule of SOA
your systems will fail
what’s your response?
17

18. 18
https://kateheddleston.com/blog/the-null-process

19. 1919
1
2
3
4

20. 20
Nygard’s circuit breaker

21. 21

22. 22

23. 23

24. 24

25. 25
Circuit Breaker States:
* Healthy (or “closed”)
* Recovering (or “half-open”)
* Unhealthy (or “open”)

26. 26

27. 27
Recovery:
* Wait for recovery_timeout seconds
* Send a trial request, trust its results

28. 28
Before a circuit breaker:

29. 29
Before a circuit breaker:
* Diners wait forever to get food

30. 30
Before a circuit breaker:
* Diners wait forever to get food
* Kitchen has a growing backlog

31. 31
Before a circuit breaker:
* Diners wait forever to get food
* Kitchen has a growing backlog
* New diners making things worse

32. 32
With a circuit breaker:

33. 33
With a circuit breaker:
* Fewer frustrated users

34. 34
With a circuit breaker:
* Fewer frustrated users
* Reduced load on the backend

35. 35
With a circuit breaker:
* Fewer frustrated users
* Reduced load on the backend
* A non-null process

36. 36

37. 37

38. 38

39. 39

40. 40

41. 41
Should our waiters all agree?
Module 1:

42. 42

43. 43

44. 44
New Behavior:
* Clients inform each other
* Processes are no longer independent

45. 45
* Propagate failure faster
* Requires distributed datastore
* Forces decisions about consistency

46. 46
What should we do in response?
Module 2:

47. 47

48. 48

49. 49

50. 50

51. 51
New Behavior:
* Code can check in advance about
healthiness of system
* Automatic monitoring!

52. 52
* Build features on top of system
health status
* Requires a single source of truth?

53. 53
Who decides we’re unhealthy?
Module 3:

54. 54

55. 55
def signal_overload(cb):
if len(jobs) > THRESH:
cb.mark_unhealthy()

56. 56
New Behavior:
* CB gets signals from anywhere
* Signal combining logic

57. 57
* Allows many (many) new signals
* Must combine signals
* Adds complexity to system

58. 58
How do we recover?
Module 4:

59. 59

60. 60
Dark launched requests:
* Reject but process normally
* Dangerous with side effects

61. 61
Synthetic Requests:
* Dark launching with fake requests
* Not necessarily representative

62. 62
Live requests:
* Definitely representative
* Exposes real users to pain

63. 63
New Behavior:
* Traffic determines health
* Removal of recovery timeouts

64. 64
* Faster(?) recovery
* No timeout tuning required
* Dark launching not always possible
* Synthetic can be unrepresentative

65. 65
in summary

66. 66
Your system will fail, have a plan!

67. 67
The basic CB is better than nothing

68. 68
Questions to ask:
* Should our waiters all agree?
* How should I deal with unhealthiness?
* Who decides we’re unhealthy?
* How do we recover?

69. 69
Questions to ask:
* Should our waiters all agree?
* How should I deal with unhealthiness?
* Who decides we’re unhealthy?
* How do we recover?

70. 70
Questions to ask:
* Should our waiters all agree?
* How should I deal with unhealthiness?
* Who decides we’re unhealthy?
* How do we recover?

71. 71
Questions to ask:
* Should our waiters all agree?
* How should I deal with unhealthiness?
* Who decides we’re unhealthy?
* How do we recover?

72. 72
Questions to ask:
* Should our waiters all agree?
* How should I deal with unhealthiness?
* Who decides we’re unhealthy?
* How do we recover?

73. 73
…and much more!

74. Much comes down to your use case
74

75. Questions?
75
striglia@yelp.com
@scott_triglia

77. 77
Can’t we do better than
rejecting requests?

78. http://techblog.netflix.com/2011/12/
making-netflix-api-more-resilient.html

79. 79
How do I safely test out a
new circuit breaker?

80. https://engineering.heroku.com/blogs/
2015-06-30-improved-production-
stability-with-circuit-breakers/