Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Building Resilient Serverless Systems with Non-Serverless Components


Published on

Serverless functions (like AWS Lambda, Google Cloud Functions, and Azure Functions) have the ability to scale almost infinitely to handle massive workload spikes. While this is a great solution to compute, it can be a MAJOR PROBLEM for other downstream resources like RDBMS, third-party APIs, legacy systems, and even most managed services hosted by your cloud provider. Whether you’re maxing out database connections, exceeding API quotas, or simply flooding a system with too many requests at once, serverless functions can DDoS your components and potentially take down your application. In this talk, we’ll discuss strategies and architectural patterns to create highly resilient serverless applications that can mitigate and alleviate pressure on non-serverless downstream systems during peak load times.

Published in: Software
  • Be the first to comment

  • Be the first to like this

Building Resilient Serverless Systems with Non-Serverless Components

  1. 1. Building Resilient Serverless Systems with Non-Serverless Components Jeremy Daly CTO, @jeremy_daly
  2. 2. Jeremy Daly • CTO at • Consultant that works with companies building in the cloud • 20+ year veteran of technology startups • Started migrating workloads to the cloud in 2009 • Blogger, open-source contributor, speaker • Publish Off-by-none, a weekly newsletter about serverless @jeremy_daly
  3. 3. Agenda • What does it mean to be Serverless? • Working with “less-than-scalable” RDBMS • Using unreliable APIs • Managing API quotas • Other non-serverless components @jeremy_daly
  4. 4. What does it mean to be Serverless? • No server management • Flexible scaling • Pay for value • Automated high availability @jeremy_daly
  5. 5. What does it mean to be Serverless? @jeremy_daly ElastiCache RDS EMR Amazon ES Redshift Fargate Anything “on EC2”Lambda Cognito Kinesis S3 DynamoDB SQS SNS API Gateway CloudWatch AppSync IoT Comprehend Serverless Managed Not Serverless DocumentDB (MongoDB) Managed Streaming for Kafka Definitely
  6. 6. @jeremy_daly
  7. 7. Everything has limits! • Reserved Concurrency 🚦 • FunctionTimeouts ⏳ • Memory Limits 🧠 • NetworkThroughput 🚰 Some components are better than others @jeremy_daly Know Your Limits
  8. 8. “I want my, I want my, I want my SQL” ~ Dire Straits
  9. 9. Simple ServerlessWeb Service Client API Gateway Lambda DynamoDB @jeremy_daly Highly Scalable Highly Scalable Highly Scalable
  10. 10. Simple ServerlessWeb Service Client API Gateway Lambda @jeremy_daly Highly Scalable Highly Scalable NotThat Scalable 😳 RDS ^ not so RDBMS and FaaS don’t play nicely together: • Concurrency model doesn’t allow connection pooling • Limited number of DB connections available • Recycled containers create zombies
  11. 11. Ways to Manage DB Connections • Increase max_connections setting • Limit concurrent executions • Lower your connection timeouts • Limit connections per username • Close connection before function ends @jeremy_daly 🤞 😡 ⚠ 🎲 😱 👎
  12. 12. BetterWays to Manage DB Connections • Implement a good caching strategy • Buffer events for throttling and durability • Manage connections ourselves @jeremy_daly 👎
  13. 13. hit Implement a good caching strategy Client API Gateway RDSLambda Elasticache Key Points: • Create new RDS connections ONLY on misses • Make sureTTLs are set appropriately • Include the ability to invalidate cache @jeremy_daly
  14. 14. Do you really need immediate feedback? • Synchronous Communication ⏳ Services can be invoked by other services and must wait for a reply. This is considered a blocking request, because the invoking service cannot finish executing until a response is received. • Asynchronous Communication 🚀 This is a non-blocking request. A service can invoke (or trigger) another service directly or it can use another type of communication channel to queue information.The service typically only needs to wait for confirmation (ack) that the request was received. @jeremy_daly
  15. 15. RDS Buffer events for throttling and durability Client API Gateway SQS Queue SQS (DLQ) Lambda Lambda (throttled) ack “Asynchronous” Request Synchronous Request @jeremy_daly Key Points: • SQS adds durability • Throttled Lambdas reduce downstream pressure • Failed events are stored for further inspection/replay Limit the concurrency to match RDS throughput
  16. 16. Manage connections ourselves 1. Count open connections 2. Close connection if connection ratio threshold exceeded 3. Close sleeping connections with high time values 4. Retry connections with exponential back off @jeremy_daly
  17. 17. Serverless MySQL @jeremy_daly
  18. 18. Count open connections @jeremy_daly Query the processlist to get the total number of active connections
  19. 19. Close connection if over ratio threshold @jeremy_daly If we exceed the connection ratio Calculate our timeout Try to kill zombies If no zombies, terminate connection Else, just try to kill zombies
  20. 20. Close sleeping connections with high time values @jeremy_daly Query processlist for zombies Kill zombies
  21. 21. Retry connections with exponential back off @jeremy_daly If error trying to connect Retry with Jitter
  22. 22. Does this really work? @jeremy_daly • Aurora Serverless (2 ACUs) • 90 connections available • 1,024 MB of memory • 500 users/sec for one minute • Avg. response time was 41 ms • ZERO ERRORS
  23. 23. We shouldn’t have to do this! @jeremy_daly Amazon Aurora Serverless Aurora Serverless DATA API Doesn’t solve the max_connections issue Still in PREVIEW, not ready for primetime
  24. 24. Third-Party APIs
  25. 25. Manage calls to third-party APIs • Implement a good caching strategy • Buffer events for throttling and durability • Implement circuit breakers @jeremy_daly
  26. 26. DynamoDB Stripe API The Circuit Breaker Client API Gateway Lambda Key Points: • Cache your cache with warm functions • Use a reasonable failure count • Understand idempotency Status Check CLOSED OPEN Increment Failure Count HALF OPEN “Everything fails all the time.” ~WernerVogels @jeremy_daly
  27. 27. What about quotas? • Concurrency has no effect on frequency • Stateless functions are not coordinated • Step Functions would be very expensive • Adding state wouldn’t prevent needless invocations @jeremy_daly
  28. 28. Can we build a better system? • 100% serverless • Cost effective • Scalable • Resilient • Efficient • Coordinated @jeremy_daly
  29. 29. Lambda Orchestrator (concurrency 1) The Lambda Orchestrator DynamoDB LambdaWorker LambdaWorker LambdaWorker Concurrent Executions of the SAME function SQS (DLQ) @jeremy_daly CloudWatch Rule (trigger every minute) SQS QueueSQS (DLQ) Status? Gmail API 250 Quota Units per minute
  30. 30. Distributing Events Key Points: • SNS has a “well-defined API” • Decouples downstream processes • Allows multiple subscribers with message filters Client SNS “Asynchronous” Request ack Event Service @jeremy_daly HTTP SMS Lambda SQS Email
  31. 31. Stripe API @jeremy_daly Distribute &Throttle ack SQS Queue Lambda (concurrency 25) SNS Topic Client API Gateway Lambda Order Service total > $0 Key Points: • SNS to SQS is “guaranteed” (100,010 retries) • Filter events to selectively trigger services • Manage throttling/quotas per service RDS SQS Queue Lambda (concurrency 10) SMS Alerting Service Twilio API SQS Queue Lambda (concurrency 5) Billing Service status == ”order_complete” Event Service
  32. 32. Other non-serverless components • Managed Services • Legacy Systems • Our own serverless APIs @jeremy_daly
  33. 33. Non-serverless components are inevitable • Know the limits of your components • Use a good caching strategy • Embrace asynchronous processes • Buffer and throttle events to distributed systems • Utilize eventual consistency @jeremy_daly
  34. 34. Things I’m working on… Blog: Newsletter: Lambda API: Podcast: GitHub: Twitter: @jeremy_daly @jeremy_daly
  35. 35. ThankYou! Jeremy Daly @jeremy_daly