Hopperx1 Seattle 2019 - Don't Let Clients Get You Down
Report
clarerichardsonFollow
Sep. 22, 2019•0 likes•71 views
Hopperx1 Seattle 2019 - Don't Let Clients Get You Down
Sep. 22, 2019•0 likes•71 views
Download to read offline
Report
Engineering
Don't Let Clients Get You Down - Configuring Fault-Tolerant Clients in Resilient Systems
clarerichardsonFollow
More Related Content
Recently uploaded
Chapter 8. Classification Basic Concepts.pptSubrata Kumer Paul
Three Phase Induction Motor.pptxNoorulAmin67
Data Communication & Computer NetworksSreedhar Chowdam
THEODOLITE.ppsxMOHDTARIQFAROOQUI
UNII II ELECTRICAL SYSTEMS IN BUILDINGS karthi keyan
Chapter 2. Know Your Data.pptSubrata Kumer Paul
Featured
Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...RachelPearson36
Unlocking the Power of ChatGPT and AI in Testing - A Real-World Look, present...Applitools
12 Ways to Increase Your Influence at WorkGetSmarter
ChatGPT webinar slidesAlireza Esmikhani
More than Just Lines on a Map: Best Practices for U.S Bike RoutesProject for Public Spaces & National Center for Biking and Walking
Ride the Storm: Navigating Through Unstable Periods / Katerina Rudko (Belka G...DevGAMM Conference
Hopperx1 Seattle 2019 - Don't Let Clients Get You Down
- 1. Don't Let Clients Get You Down Configuring Fault-Tolerant Clients in Resilient Systems Clare Liguori Principal Engineer, Amazon Web Services #Hopperx1Seattle
- 4. LB
- 5. LB =(
- 6. LB =(
- 7. LB =(
- 8. X X LB =(
- 9. Retry #Hopperx1Seattle Timeout Backoff Don’t Let Clients Get You Down
- 10. Retry #Hopperx1Seattle Timeout Backoff Don’t Let Clients Get You Down
- 11. LB =( If at first you don’t succeed
- 12. LB =( Retry!
- 13. $ npm install npm ERR! fetch failed git+ssh://...
- 15. LB =( =( =( 10 Retries = 100 Requests x 10 Retries Set sane retry limits!
- 16. LB =( Only retry on idempotent APIs Pizzas ordered: 0 =(
- 17. LB =( Only retry on idempotent APIs Pizzas ordered: 0 =( Try #1: Connection drops
- 18. LB =( Only retry on idempotent APIs Pizzas ordered: 0 =( Try #2: Connection drops
- 19. LB =( Only retry on idempotent APIs Pizzas ordered: 1 =( Try #3: Success!
- 20. LB Only retry on idempotent APIs Pizzas ordered: 3 Uh oh, those servers are healthy again
- 21. Retry #Hopperx1Seattle Timeout Backoff Don’t Let Clients Get You Down
- 22. LB
- 23. connection timeout socket timeout read timeout write timeout individual request timeout overall timeout including retries
- 26. $ curl --retry 3 --connect-timeout 10 --max-time 20 https://checkip.amazonaws.com
- 27. Retry #Hopperx1Seattle Timeout Backoff Don’t Let Clients Get You Down
- 28. LB =( =( =( 10 Retries = 100 Requests x 10 Retries
- 29. sleep(2)
- 35. $ npm install retry-cli $ retry –-retries=3 --factor=2 --randomize -- curl --connect-timeout 10 https://checkip.amazonaws.com
- 36. $ npm install retry-cli $ retry –-retries=3 --factor=2 --randomize -- curl --connect-timeout 10 https://checkip.amazonaws.com Retry Exponential Backoff Backoff Jitter Timeout
- 37. Retry #Hopperx1Seattle Timeout Backoff Don’t Let Clients Get You Down
Editor's Notes
- Intro: I’m Clare Why do clients matter to resiliency?
- Example multi-tier system with microservices
- There are lots of clients in this system
- Drill into service: load balancer + multiple servers
- What happens when a single server has problems?
- Hopefully something like health checks realizes its down and removes it from load balancer
- BUT until then... clients are still going to that server!
- And the failure cascades all the way up the system! So, how can we configure clients to be resilient to failures, so that our entire system is resilient?
- Today we’ll talk about three important aspects of client resiliency
- Let’s start with retries
- Retries are self-explanatory
- Retrying likely gets request directed to healthy node
- Real world example: Installing npm modules during a cron job
- Enable retries in SSH
- Set retry limits (3-5), don’t make a bad situation worse
- What happens when an API doesn’t de-duplicate requests?
- For example, if a network connection drops, you don’t have confirmation of whether the request is still alive on that server
- The servers may have just been processing requests slowly, but eventually complete them. Idempotent APIs will have a field like “idempotency token” or “retry token” where you can provide a token that signifies only ONE request with this token should be processed.
- Timeouts give opportunity to retry in case of networking or load problems where requests are very slow. Timeouts also ensure slow requests don't hog all your threads
- Lots of different types of timeouts, varies by client library how much control you get over each of these
- Favorite is socket read timeout: many systems have a default timeout of infinity (FOREVER!). Can result in hung threads that stack up over time
- Real world example: curl script would randomly hang forever
- Configure timeouts and retries
- Too many retries can get you throttled, so backoff lets you retry only the number of times you need to
- Simplest backoff, so you’re not hammering the servers
- Better: exponential backoff: delay more every time you retry
- What happens if every client backs off the same amount?
- They all cause high load at the same time
- Add jitter: each client randomizes the amount they delay a little bit
- Smoothes out the request load
- Use retry-cli to get exponential backoff + randomized delay (i.e. jitter)
- Use retry-cli to get exponential backoff + randomized delay (i.e. jitter)
- We’ve covered retries, timeouts, and backoffs. Go inspect your clients!
- Questions?