Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Don't Let Clients Get You Down
Configuring Fault-Tolerant Clients in Resilient Systems
Clare Liguori
Principal Engineer, A...
Clients! Clients!
LB
LB
=(
LB
=(
LB
=(
X
X
LB
=(
Retry
#Hopperx1Seattle
Timeout Backoff
Don’t Let Clients Get You Down
Retry
#Hopperx1Seattle
Timeout Backoff
Don’t Let Clients Get You Down
LB
=(
If at first
you don’t
succeed
LB
=(
Retry!
$ npm install
npm ERR! fetch failed git+ssh://...
~/.ssh/config:
Host *
ConnectionAttempts 3
LB
=(
=(
=(
10 Retries
=
100 Requests
x
10 Retries
Set sane
retry limits!
LB
=(
Only retry on
idempotent APIs
Pizzas
ordered:
0
=(
LB
=(
Only retry on
idempotent APIs
Pizzas
ordered:
0
=(
Try #1:
Connection
drops
LB
=(
Only retry on
idempotent APIs
Pizzas
ordered:
0
=(
Try #2:
Connection
drops
LB
=(
Only retry on
idempotent APIs
Pizzas
ordered:
1
=(
Try #3:
Success!
LB
Only retry on
idempotent APIs
Pizzas
ordered:
3
Uh oh, those
servers are
healthy again
Retry
#Hopperx1Seattle
Timeout Backoff
Don’t Let Clients Get You Down
LB
connection timeout
socket timeout
read timeout
write timeout
individual request timeout
overall timeout including retries
Time
ThreadsInUse
=(
=)
Max Threads
$ curl https://checkip.amazonaws.com
$ curl --retry 3 
--connect-timeout 10 
--max-time 20 
https://checkip.amazonaws.com
Retry
#Hopperx1Seattle
Timeout Backoff
Don’t Let Clients Get You Down
LB
=(
=(
=(
10 Retries
=
100 Requests
x
10 Retries
sleep(2)
sleep(0.5)
sleep(1)
sleep(2)
sleep(4)
sleep(8)
sleep(0.5)
sleep(1)
sleep(2)
sleep(4)
sleep(8)
sleep(0.5)
sleep(1)
sleep(2)
sleep(4)
sleep(8)
sleep(0.5)
sleep(1)
sleep(2)...
Time
Requests
=(
=)
=(
=(
=(
=(
=(
=)
=)
=)
=)
sleep(0.6)
sleep(1.1)
sleep(2.1)
sleep(4.1)
sleep(8.1)
sleep(0.7)
sleep(1.2)
sleep(2.2)
sleep(4.2)
sleep(8.2)
sleep(0.8)
s...
Time
Requests
=)
=)
=)
=)
=)
=)
=)
=)
=)
=)
=)
=)
=)
=)
=)
=)
=)
=)
=)
=)
=)
=)
$ npm install retry-cli
$ retry –-retries=3 
--factor=2 
--randomize -- 
curl --connect-timeout 10 
https://checkip.amazon...
$ npm install retry-cli
$ retry –-retries=3 
--factor=2 
--randomize -- 
curl --connect-timeout 10 
https://checkip.amazon...
Retry
#Hopperx1Seattle
Timeout Backoff
Don’t Let Clients Get You Down
Thank you!
#Hopperx1Seattle
Hopperx1 Seattle 2019 - Don't Let Clients Get You Down
You’ve finished this document.
Download and read it offline.
Upcoming SlideShare
What to Upload to SlideShare
Next
Upcoming SlideShare
What to Upload to SlideShare
Next
Download to read offline and view in fullscreen.

0

Share

Hopperx1 Seattle 2019 - Don't Let Clients Get You Down

Download to read offline

Don't Let Clients Get You Down - Configuring Fault-Tolerant Clients in Resilient Systems

Related Books

Free with a 30 day trial from Scribd

See all

Related Audiobooks

Free with a 30 day trial from Scribd

See all
  • Be the first to like this

Hopperx1 Seattle 2019 - Don't Let Clients Get You Down

  1. 1. Don't Let Clients Get You Down Configuring Fault-Tolerant Clients in Resilient Systems Clare Liguori Principal Engineer, Amazon Web Services #Hopperx1Seattle
  2. 2. Clients! Clients!
  3. 3. LB
  4. 4. LB =(
  5. 5. LB =(
  6. 6. LB =(
  7. 7. X X LB =(
  8. 8. Retry #Hopperx1Seattle Timeout Backoff Don’t Let Clients Get You Down
  9. 9. Retry #Hopperx1Seattle Timeout Backoff Don’t Let Clients Get You Down
  10. 10. LB =( If at first you don’t succeed
  11. 11. LB =( Retry!
  12. 12. $ npm install npm ERR! fetch failed git+ssh://...
  13. 13. ~/.ssh/config: Host * ConnectionAttempts 3
  14. 14. LB =( =( =( 10 Retries = 100 Requests x 10 Retries Set sane retry limits!
  15. 15. LB =( Only retry on idempotent APIs Pizzas ordered: 0 =(
  16. 16. LB =( Only retry on idempotent APIs Pizzas ordered: 0 =( Try #1: Connection drops
  17. 17. LB =( Only retry on idempotent APIs Pizzas ordered: 0 =( Try #2: Connection drops
  18. 18. LB =( Only retry on idempotent APIs Pizzas ordered: 1 =( Try #3: Success!
  19. 19. LB Only retry on idempotent APIs Pizzas ordered: 3 Uh oh, those servers are healthy again
  20. 20. Retry #Hopperx1Seattle Timeout Backoff Don’t Let Clients Get You Down
  21. 21. LB
  22. 22. connection timeout socket timeout read timeout write timeout individual request timeout overall timeout including retries
  23. 23. Time ThreadsInUse =( =) Max Threads
  24. 24. $ curl https://checkip.amazonaws.com
  25. 25. $ curl --retry 3 --connect-timeout 10 --max-time 20 https://checkip.amazonaws.com
  26. 26. Retry #Hopperx1Seattle Timeout Backoff Don’t Let Clients Get You Down
  27. 27. LB =( =( =( 10 Retries = 100 Requests x 10 Retries
  28. 28. sleep(2)
  29. 29. sleep(0.5) sleep(1) sleep(2) sleep(4) sleep(8)
  30. 30. sleep(0.5) sleep(1) sleep(2) sleep(4) sleep(8) sleep(0.5) sleep(1) sleep(2) sleep(4) sleep(8) sleep(0.5) sleep(1) sleep(2) sleep(4) sleep(8) sleep(0.5) sleep(1) sleep(2) sleep(4) sleep(8)
  31. 31. Time Requests =( =) =( =( =( =( =( =) =) =) =)
  32. 32. sleep(0.6) sleep(1.1) sleep(2.1) sleep(4.1) sleep(8.1) sleep(0.7) sleep(1.2) sleep(2.2) sleep(4.2) sleep(8.2) sleep(0.8) sleep(1.3) sleep(2.3) sleep(4.3) sleep(8.3) sleep(0.9) sleep(1.4) sleep(2.4) sleep(4.4) sleep(8.4) Add jitter!
  33. 33. Time Requests =) =) =) =) =) =) =) =) =) =) =) =) =) =) =) =) =) =) =) =) =) =)
  34. 34. $ npm install retry-cli $ retry –-retries=3 --factor=2 --randomize -- curl --connect-timeout 10 https://checkip.amazonaws.com
  35. 35. $ npm install retry-cli $ retry –-retries=3 --factor=2 --randomize -- curl --connect-timeout 10 https://checkip.amazonaws.com Retry Exponential Backoff Backoff Jitter Timeout
  36. 36. Retry #Hopperx1Seattle Timeout Backoff Don’t Let Clients Get You Down
  37. 37. Thank you! #Hopperx1Seattle

Don't Let Clients Get You Down - Configuring Fault-Tolerant Clients in Resilient Systems

Views

Total views

37

On Slideshare

0

From embeds

0

Number of embeds

0

Actions

Downloads

1

Shares

0

Comments

0

Likes

0

×