Patterns and practices for building resilient Serverless applications

Patterns and practices for building resilient Serverless applications
Patterns and Practices for building resilient serverless applications presented by Yan Cui @theburningmonk
@theburningmonk theburningmonk.com
@theburningmonk theburningmonk.com “the capacity to recover quickly from difficulties; toughness.” resilience /rɪˈzɪlɪəns/ noun
@theburningmonk theburningmonk.com “the capacity to recover quickly from difficulties; toughness.” resilience /rɪˈzɪlɪəns/ noun it’s not about preventing failures!
everything fails, all the time
@theburningmonk theburningmonk.com we need to build applications that can withstand failures
@theburningmonk theburningmonk.com don’t run your application on one server…
@theburningmonk theburningmonk.com entire data centers can go down…
@theburningmonk theburningmonk.com run your application in multiple AZs and regions
@theburningmonk theburningmonk.com Failures on load: exhaustion of resources
@theburningmonk theburningmonk.com Failures on load: exhaustion of resources
@theburningmonk theburningmonk.com latency reqs/s Failures on load: exhaustion of resources CPU saturation
@theburningmonk theburningmonk.com Failures in distributed systems Service A Service B Service C user
@theburningmonk theburningmonk.com Failures in distributed systems Service A Service B Service C user
@theburningmonk theburningmonk.com Failures in distributed systems Service A Service B Service C user HTTP 502
@theburningmonk theburningmonk.com Failures in distributed systems Service A Service B Service C user You suck!
@theburningmonk theburningmonk.com microservices death stars circa 2015
Yan Cui http://theburningmonk.com @theburningmonk AWS user for 10 years
Yan Cui http://theburningmonk.com @theburningmonk http://bit.ly/yubl-serverless
Yan Cui http://theburningmonk.com @theburningmonk Developer Advocate @
Yan Cui http://theburningmonk.com @theburningmonk Independent Consultant advisetraining delivery
by Uwe Friedrichsen
@theburningmonk theburningmonk.com Lambda execution environment
@theburningmonk theburningmonk.com Serverless - multiple AZ’s out of the box Total resources created: 1 API Gateway 1 Lambda
@theburningmonk theburningmonk.com Serverless - multiple AZ’s out of the box Total resources created: 1 API Gateway 1 Lambda don’t pay for idle redundant resources!
@theburningmonk theburningmonk.com Load balancing
@theburningmonk theburningmonk.com Data replication in different AZ’s DynamoDB Global Tables
@theburningmonk theburningmonk.com There are throttling everywhere!
@theburningmonk theburningmonk.com Beware of timeout mismatch API Gateway  Integration timeout   Default: 29s Lambda  Timeout Max: 15 minutes
@theburningmonk theburningmonk.com Beware of timeout mismatch Lambda  Timeout Max: 15 minutes SQS  Visibility timeout  Default: 30s Min: 0s Max: 12 hours
@theburningmonk theburningmonk.com Beware of timeout mismatch Lambda  Timeout Max: 15 minutes SQS  Visibility timeout  Default: 30s Min: 0s Max: 12 hours set VisibilityTimeout to 6x Lambda timeout
@theburningmonk theburningmonk.com Offload computing operations to queues
@theburningmonk theburningmonk.com Offload computing operations to queues
@theburningmonk theburningmonk.com Offload computing operations to queues better absorb downstream problems
@theburningmonk theburningmonk.com Offload computing operations to queues need way to replay DLQ events
https://www.npmjs.com/package/lumigo-cli
@theburningmonk theburningmonk.com Offload computing operations to queues great for fire-and-forget tasks
@theburningmonk theburningmonk.com “what if the client is waiting for a response?”
@theburningmonk theburningmonk.com “Decoupled Invocation”
@theburningmonk theburningmonk.com task id created at result xxx xxx <null> xxx xxx <null> … … … task results not ready…
@theburningmonk theburningmonk.com task id created at result xxx xxx <null> xxx xxx <null> … … … task results not ready… 202
@theburningmonk theburningmonk.com task id created at result xxx xxx <null> xxx xxx <null> … … … task results reporting for duty!
@theburningmonk theburningmonk.com task id created at result xxx xxx <null> xxx xxx <null> … … … task results working hard… not ready…
@theburningmonk theburningmonk.com task id created at result xxx xxx <null> xxx xxx <null> … … … task results 202 working hard…
@theburningmonk theburningmonk.com task id created at result xxx xxx <null> xxx xxx { … } … … … task results done!
@theburningmonk theburningmonk.com task id created at result xxx xxx <null> xxx xxx { … } … … … task results done!
@theburningmonk theburningmonk.com task id created at result xxx xxx <null> xxx xxx { … } … … … task results 200 { … }
@theburningmonk theburningmonk.com wait…
@theburningmonk theburningmonk.com a distributed transaction!
@theburningmonk theburningmonk.com a distributed transaction! needs rollback
@theburningmonk theburningmonk.com no distributed transactions
@theburningmonk theburningmonk.com do the work here
@theburningmonk theburningmonk.com retry-until-success
@theburningmonk theburningmonk.com 24 hours data retention
@theburningmonk theburningmonk.com 24 hours data retention need alerting to ensure issue are addressed quickly
@theburningmonk theburningmonk.com retry-until-success needs to deal with poinson messages
@theburningmonk theburningmonk.com what if you can’t avoid distributed transactions?
@theburningmonk theburningmonk.com The Saga pattern A pattern for managing failures where each action has a compensating action for rollback
@theburningmonk theburningmonk.com The Saga pattern https://www.youtube.com/watch?v=xDuwrtwYHu8
@theburningmonk theburningmonk.com The Saga pattern Begin transaction Start book hotel request End book hotel request Start book flight request End book flight request Start book car rental request End book car rental request End transaction
@theburningmonk theburningmonk.com The Saga pattern model both actions and compensating actions as Lambda functions
@theburningmonk theburningmonk.com The Saga pattern use Step Functions as the coordinator for the saga
@theburningmonk theburningmonk.com The Saga pattern Input
@theburningmonk theburningmonk.com The Saga pattern
@theburningmonk theburningmonk.com retry-until-success needs to deal with poinson messages Mind the poison message
@theburningmonk theburningmonk.com Mind the poison message
@theburningmonk theburningmonk.com Mind the poison message 6, 3, 1, 1, 1, 1, …
@theburningmonk theburningmonk.com Mind the poison message 6, 3, 1, 1, 1, 1, … only count the “same” batch
@theburningmonk theburningmonk.com Mind the poison message have to fetch from the stream
@theburningmonk theburningmonk.com Mind the poison message have to fetch from the stream do it before they expire from the stream!
@theburningmonk theburningmonk.com how do you prevent building up an insurmountable backlog?
@theburningmonk theburningmonk.com Load shedding implement load shedding prioritize newer messages with a better chance to succeed
@theburningmonk theburningmonk.com Load shedding excess load is sent to DLQ
@theburningmonk theburningmonk.com Load shedding process with a delay
@theburningmonk theburningmonk.com Mind the partial failures LambdaSQS
@theburningmonk theburningmonk.com Mind the partial failures LambdaSQS Poller
@theburningmonk theburningmonk.com LambdaSQS Poller Mind the partial failures Delete
@theburningmonk theburningmonk.com Mind the partial failures LambdaSQS Poller Error
@theburningmonk theburningmonk.com Mind the partial failures LambdaSQS Poller Error DLQ
@theburningmonk theburningmonk.com Mind the partial failures LambdaSQS Poller Error DLQ batch fails as a unit
https://lumigo.io/blog/sqs-and-lambda-the-missing-guide-on-failure-modes Mind the partial failures
@theburningmonk theburningmonk.com Mind the partial failures
@theburningmonk theburningmonk.com Mind the retry storm Service A
@theburningmonk theburningmonk.com Mind the retry storm Service A retry retry retry retry
@theburningmonk theburningmonk.com retry storm
@theburningmonk theburningmonk.com circuit breaker pattern After X consecutive timeouts, trip the circuit
@theburningmonk theburningmonk.com circuit breaker pattern After X consecutive timeouts, trip the circuit When circuit is open, fail fast
@theburningmonk theburningmonk.com circuit breaker pattern When circuit is open, fail fast but, allow 1 request through every Y mins After X consecutive timeouts, trip the circuit
@theburningmonk theburningmonk.com circuit breaker pattern When circuit is open, fail fast but, allow 1 request through every Y mins If request succeeds, close the circuit After X consecutive timeouts, trip the circuit
@theburningmonk theburningmonk.com where do I keep the state of the circuit?
@theburningmonk theburningmonk.com in-memory Service A isOpen: false isOpen: false isOpen: false isOpen: false
@theburningmonk theburningmonk.com in-memory Service A isOpen: true isOpen: false isOpen: true isOpen: false
@theburningmonk theburningmonk.com in-memory PROS simplicity
@theburningmonk theburningmonk.com in-memory PROS simplicity no dependency on external service requires another circuit breaker to protect… cost & maintenance overhead (IAM, infra, etc.)
@theburningmonk theburningmonk.com in-memory PROS simplicity no dependency on external service CONS takes longer & more requests to stop all traffic
@theburningmonk theburningmonk.com in-memory PROS simplicity no dependency on external service CONS takes longer & more requests to stop all traffic new containers would generate more traffic
@theburningmonk theburningmonk.com external service Service AisOpen: false
@theburningmonk theburningmonk.com external service Service AisOpen: true
@theburningmonk theburningmonk.com external service Service AisOpen: true
@theburningmonk theburningmonk.com external service PROS minimizes no. of total requests to trip circuit new containers respect collective decision CONS complexity dependency on an external service
@theburningmonk theburningmonk.com which approach should I use?
@theburningmonk theburningmonk.com which approach should I use? It depends. Maybe start with the simplest solution first?
@theburningmonk theburningmonk.com Lambda autoscaling Burst concurrency limits:  3000 – US West (Oregon), US East (N. Virginia), Europe (Ireland), 1000 – Asia Pacific (Tokyo), Europe (Frankfurt), 500 – Other Regions Burst: 500 new instances / each minute 
@theburningmonk theburningmonk.com Lambda autoscaling Burst concurrency limits:  3000 – US West (Oregon), US East (N. Virginia), Europe (Ireland), 1000 – Asia Pacific (Tokyo), Europe (Frankfurt), 500 – Other Regions Burst: 500 new instances / each minute  Standard burst concurrency limits when over the provisioned capacity  
@theburningmonk theburningmonk.com Lambda autoscaling Burst concurrency limits:  3000 – US West (Oregon), US East (N. Virginia), Europe (Ireland), 1000 – Asia Pacific (Tokyo), Europe (Frankfurt), 500 – Other Regions Burst: 500 new instances / each minute  Adjustable provisioned capacity based on CloudWatch metrics Standard burst concurrency limits when over the provisioned capacity  
@theburningmonk theburningmonk.com Lambda limitations & throttling Concurrent executions: 1000*  Timeout: 15 minutes  Burst concurrency: 500 - 3000  Burst: 500 new instances / minute * Can be increased with support ticket
@theburningmonk theburningmonk.com Lambda limitations & throttling good for spikey traffic, up to a point Concurrent executions: 1000*  Timeout: 15 minutes  Burst concurrency: 500 - 3000  Burst: 500 new instances / minute * Can be increased with support ticket
@theburningmonk theburningmonk.com “what if my traffic is more spiky than that?”
@theburningmonk theburningmonk.com Scenario: predictable spikes Holidays, weekends,  celebrations  (Black Friday) Planned launch of  resources  (new series available) Sport events
@theburningmonk theburningmonk.com Scenario: predictable spikes scheduled auto-scaling
@theburningmonk theburningmonk.com Scenario: predictable spikes scheduled auto-scaling the burst limits still apply, factor the timing into account
@theburningmonk theburningmonk.com Scenario: predictable spikes
@theburningmonk theburningmonk.com Scenario: unpredictable spikes Traffic generated by user actions    Jennifer Aniston’s first post
@theburningmonk theburningmonk.com “if Lambda scaling is the problem…”
@theburningmonk theburningmonk.com Client only needs an acknowledgement

Check these out next

Patterns and practices for building resilient Serverless applications

Recommended

More Related Content

Slideshows for you(20)

Similar to Patterns and practices for building resilient Serverless applications(20)

More from Yan Cui(20)

Recently uploaded(20)

Patterns and practices for building resilient Serverless applications