Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Hopperx1 Seattle 2019 - Don't Let Clients Get You Down

5 views

Published on

Don't Let Clients Get You Down - Configuring Fault-Tolerant Clients in Resilient Systems

Published in: Engineering
  • Be the first to comment

  • Be the first to like this

Hopperx1 Seattle 2019 - Don't Let Clients Get You Down

  1. 1. Don't Let Clients Get You Down Configuring Fault-Tolerant Clients in Resilient Systems Clare Liguori Principal Engineer, Amazon Web Services #Hopperx1Seattle
  2. 2. Clients! Clients!
  3. 3. LB
  4. 4. LB =(
  5. 5. LB =(
  6. 6. LB =(
  7. 7. X X LB =(
  8. 8. Retry #Hopperx1Seattle Timeout Backoff Don’t Let Clients Get You Down
  9. 9. Retry #Hopperx1Seattle Timeout Backoff Don’t Let Clients Get You Down
  10. 10. LB =( If at first you don’t succeed
  11. 11. LB =( Retry!
  12. 12. $ npm install npm ERR! fetch failed git+ssh://...
  13. 13. ~/.ssh/config: Host * ConnectionAttempts 3
  14. 14. LB =( =( =( 10 Retries = 100 Requests x 10 Retries Set sane retry limits!
  15. 15. LB =( Only retry on idempotent APIs Pizzas ordered: 0 =(
  16. 16. LB =( Only retry on idempotent APIs Pizzas ordered: 0 =( Try #1: Connection drops
  17. 17. LB =( Only retry on idempotent APIs Pizzas ordered: 0 =( Try #2: Connection drops
  18. 18. LB =( Only retry on idempotent APIs Pizzas ordered: 1 =( Try #3: Success!
  19. 19. LB Only retry on idempotent APIs Pizzas ordered: 3 Uh oh, those servers are healthy again
  20. 20. Retry #Hopperx1Seattle Timeout Backoff Don’t Let Clients Get You Down
  21. 21. LB
  22. 22. connection timeout socket timeout read timeout write timeout individual request timeout overall timeout including retries
  23. 23. Time ThreadsInUse =( =) Max Threads
  24. 24. $ curl https://checkip.amazonaws.com
  25. 25. $ curl --retry 3 --connect-timeout 10 --max-time 20 https://checkip.amazonaws.com
  26. 26. Retry #Hopperx1Seattle Timeout Backoff Don’t Let Clients Get You Down
  27. 27. LB =( =( =( 10 Retries = 100 Requests x 10 Retries
  28. 28. sleep(2)
  29. 29. sleep(0.5) sleep(1) sleep(2) sleep(4) sleep(8)
  30. 30. sleep(0.5) sleep(1) sleep(2) sleep(4) sleep(8) sleep(0.5) sleep(1) sleep(2) sleep(4) sleep(8) sleep(0.5) sleep(1) sleep(2) sleep(4) sleep(8) sleep(0.5) sleep(1) sleep(2) sleep(4) sleep(8)
  31. 31. Time Requests =( =) =( =( =( =( =( =) =) =) =)
  32. 32. sleep(0.6) sleep(1.1) sleep(2.1) sleep(4.1) sleep(8.1) sleep(0.7) sleep(1.2) sleep(2.2) sleep(4.2) sleep(8.2) sleep(0.8) sleep(1.3) sleep(2.3) sleep(4.3) sleep(8.3) sleep(0.9) sleep(1.4) sleep(2.4) sleep(4.4) sleep(8.4) Add jitter!
  33. 33. Time Requests =) =) =) =) =) =) =) =) =) =) =) =) =) =) =) =) =) =) =) =) =) =)
  34. 34. $ npm install retry-cli $ retry –-retries=3 --factor=2 --randomize -- curl --connect-timeout 10 https://checkip.amazonaws.com
  35. 35. $ npm install retry-cli $ retry –-retries=3 --factor=2 --randomize -- curl --connect-timeout 10 https://checkip.amazonaws.com Retry Exponential Backoff Backoff Jitter Timeout
  36. 36. Retry #Hopperx1Seattle Timeout Backoff Don’t Let Clients Get You Down
  37. 37. Thank you! #Hopperx1Seattle

×