Surviving Black Friday - CodeMotion
•
1 like•125 views
The ‘Black Friday fail’ is the greatest fear of every major online retailer. Since downtime equals money, and in Black Friday it means quite a lot of money. But the sad truth is that a failure of a service is inevitable, and not only on black Friday. So how can we survive a failure of a service when it inevitably fails? In this lecture I will share our approach for SRE. Why we all have misconceptions on how a major website failure unfolds. and how to use tools like chaos testing, gradual rollout, circuit breakers and automatic fallback to protect your system.
Recommended
More Related Content
Similar to Surviving Black Friday - CodeMotion
Similar to Surviving Black Friday - CodeMotion (20)
Recently uploaded
Recently uploaded (20)
Surviving Black Friday - CodeMotion
- 1. Surviving Black Friday Omri Fima ROME - APRIL 13/14 2018
- 2. SURVIVING BLACK FRIDAYA STORY ABOUT SHOPPING & SOFTWARE ENGINEERING
- 9. I’m not going to talk about that kind of Surviving Black Friday
- 17. #BlackFridayFail
- 18. Omri Fima • Technical Lead at Data & ML Group • Resilience engineer • Maker
- 19. Social E-Commerce We give our members a place to connect, discover, and shop with people who share their passion.
- 21. My Story
- 23. Well, not really
- 24. It’s probably more like this
- 26. or more accurately, like this
- 27. Long GC Load Balancer cluster cluster Table lockSlow API Server in japan Limited network CDN
- 28. So lets see what happens when a server fails
- 30. Or in real life
- 32. But even this is not what happens most of times.
- 33. Usually it’s more like this
- 34. Under Load
- 35. Under Load
- 38. Oy Vey…
- 39. Bazillion servers on the cloud will not make it better. Under certain scenarios it might even make it worse. Sad Truth #1
- 40. It won’t happen only in black Friday. Other good scenarios Good marketing campaign, Bad code, API load, Failure in external services, Hurricanes. Sad Truth #2
- 41. Even 99.99% reliability on 30 servers is 2HR of downtime per month Sad Truth #3
- 42. 2 Hours on Black Friday can be worth millions of $
- 43. Failur e
- 44. You can accept it or...
- 47. our job is to stop failure from cascading throughout the system
- 50. OK, nice but where do I start?
- 51. Just for you I have …
- 52. THE 7 STEP PROGRAM FOR SYSTEM RESILIENCE
- 54. First you need to understand your SLA What are our most important flows? What it means when they fail? How much can we allow them to fail?
- 55. Your page
- 57. Your “most essential” page
- 58. We can agree on SLA? GREAT! Because this is usually the hardest part.
- 60. You don’t know if a service is fault tolerant if you don’t test for faults
- 61. Simulate and test Hydra – simulate service latency and error JMETER - for testing our system under load.
- 62. Why not to just kill machines? Dead machines are The least interesting.
- 63. Why under load? Failure is a game of percentiles.
- 64. Why under load? Latency failures usually cascade under stress.
- 65. Hydra + JMETER = Great fun
- 66. Now we know what to solve first, and can easily re-test it!
- 67. Let’s pause for a second
- 68. Focus
- 72. Preventing latency cascade through the system.
- 76. Dependency A Dependency B Dependency C Dependency D UserRequest UserRequest UserRequest
- 77. Dependency A Dependency B Dependency C Dependency D UserRequest UserRequest UserRequest
- 78. Dependency A Dependency B Dependency C Dependency D UserRequest UserRequest UserRequest UserRequest UserRequest UserRequest UserRequest UserRequest UserRequest UserRequest UserRequest UserRequest UserRequest UserRequest UserRequest UserRequest
- 80. What is a good number for a timeout?
- 81. it depends
- 82. I’m OK with 0.05% of my users getting timeouts
- 83. Limit queues
- 84. Bulkhead pattern 20 Threads 10 Threads 15 Threads UserRequest UserRequest UserRequest Dependency A Dependency B Dependency C Dependency D 5 Threads Command A Command B Command C Command D
- 85. Ok, so now we are protecting our servers while annoying our users.
- 87. Remember this?
- 88. Fail Silently
- 89. A little bit too aggressive…
- 90. In the real world
- 91. Degrade to less accurate service
- 93. Use stale data from cache
- 95. Defer writes
- 96. And if I cannot fail silently?
- 97. At least fail gracefully
- 99. What is not failing gracefully?
- 100. What is not failing gracefully?
- 101. What is not failing gracefully?
- 102. What is not failing gracefully?
- 103. STEP #5 Turn broken stuff off.
- 104. Feature Toggles • Have the option to kill your features when needed. • Have the option to return your feature back to life gradually. 0% 50% 100%
- 105. Gradual rollout AB Testing Targeting Smart bulkheads. Added Bonus
- 106. STEP #6 Turn broken stuff off. Automatically.
- 107. Ok, so we have timeout of 100ms. The day we were all afraid of has come. And it works! Nobody waits more than 100ms! But everybody waits 100ms. And fail
- 108. You knew you going to fail! So why wasting everybody time on trying?
- 111. Break if you are likely to “burn the house”
- 112. Tools • Netflix Hystrix (Java) • Polly & TPL (.Net) • HystrixJS (Node) • CircuitBreakerJS (Node) • Dyno (Python)
- 114. Measure everything • when you have failed. • latency of dependencies. • when fallbacks were triggered. • when fallbacks were “almost” triggered. • when a service failure has “leaked” to the user.
- 115. Visualize everything
- 116. Tools we are using • StatsD – statistical analysis on metrics • Graphite – graphing metrics • Graphana – graphing just about anything • Skyline – alert on anomalies on grpahs. • Oculus - find relations between graphs.
- 118. Sounds trivial
- 119. Not so much. Measuring everything is easy clearing the noise is hard.
- 120. But guess what? clearing the noise makes the system more fault tolerant!
- 122. Failure is inevitabl
- 123. • Know your priorities • Simulate, Test, Fix, Repeat. • Fail Fast • Fail Silent • Turn broken stuff off • Turn broken stuff off (automatically) • Measure everything
- 124. THANK YOU MORE ABOUT US @ SEARS.CO.IL