How to debug slow lambda response times
Sep. 12, 2020•0 likes•316 views
Download to read offline
Report
Technology
One of the most common performance issues in serverless architectures is elevated latencies from external services, such as DynamoDB, ElasticSearch or Stripe. In this webinar, we will show you how to quickly identify and debug these problems, and some best practices for dealing with poor performing 3rd party services.
Yan CuiFollow
Recommended
The Need for Async @ ScalaWorldKonrad Malawski
Samuele Resca - REACTIVE PROGRAMMING, DAMN. IT IS NOT ABOUT REACTJS - Codemot...Codemotion
Reactive Programming in Java 8 with Rx-JavaKasun Indrasiri
Not Only Streams for Akademia JLabsKonrad Malawski
![[Tokyo Scala User Group] Akka Streams & Reactive Streams (0.7)](https://cdn.slidesharecdn.com/ss_thumbnails/2014-akka-streams-tokyo-140914201812-phpapp01-thumbnail.jpg?width=384&height=256&fit=bounds)
[Tokyo Scala User Group] Akka Streams & Reactive Streams (0.7)Konrad Malawski
Akka-chan's Survival Guide for the Streaming WorldKonrad Malawski
More Related Content
What's hot
The Cloud-natives are RESTless @ JavaOneKonrad Malawski
Devoxx France: Fault tolerant microservices on the JVM with CassandraChristopher Batey
Reactive java - Reactive Programming + RxJavaNexThoughts Technologies
Load Testing with RedLine13: Or getting paid to DoS your own systemsJason Lotito
utPLSQL: Unit Testing for Oracle PL/SQLSteven Feuerstein
So You Want to Write an ExporterBrian Brazil
Similar to How to debug slow lambda response times
Yan Cui - Serverless in production, an experience report - Codemotion Milan 2017Codemotion
Serverless in production, an experience report (codemotion milan)Yan Cui
AWS Lambda from the trenches (Serverless London)Yan Cui
Serverless in production (O'Reilly Software Architecture)Yan Cui
ITSubbotik - как скрестить ежа с ужом или подводные камни внедрения функциона...Vyacheslav Lapin
Open Source XMPP for Cloud Servicesmattjive
More from Yan Cui
How to win the game of trade-offsYan Cui
How to choose the right messaging serviceYan Cui
How to choose the right messaging service for your workloadYan Cui
Patterns and practices for building resilient serverless applications.pdfYan Cui
Lambda and DynamoDB best practicesYan Cui
Lessons from running AppSync in prodYan Cui
Recently uploaded
alfred-product-research-proposal.pdfAlfredSuratos
"Software Architecture for Humans!", Eberhard Wolff Fwdays
Google cloud Study Jam 2023.pptxGDSCNiT
CamundaCon NYC 2023 Keynote - Shifting into overdrive with process orchestrationBernd Ruecker
The Flutter Job Market At The MomentAhmed Abu Eldahab
Product Research PresentationDeahJadeArellano
How to debug slow lambda response times
- 1. How to debug slow Lambda response times Yan Cui @theburningmonk Developer Advocate, Lumigo AWS Serverless Hero Author of Production-Ready Serverless
- 3. Lambda autoscales by traffic
- 6. overloaded servers are a thing of the past
- 9. observation majority of performance problems originates from a function’s integration points
- 10. macro how well is this service performing in general? micro why did this transaction perform poorly?
- 11. macro micro identify systemic issues how well is this service performing in general? why did this transaction perform poorly?
- 12. how well is this service performing in general? why did this transaction perform poorly? macro micro why did this user get a bad exp?
- 13. In control theory, observability is a measure of how well internal states of a system can be inferred from knowledge of its external outputs.
- 14. In control theory, observability is a measure of how well internal states of a system can be inferred from knowledge of its external outputs. what do we need to collect?
- 15. Yan Cui http://theburningmonk.com @theburningmonk Developer Advocate @ AWS user since 2009
- 16. Yan Cui http://theburningmonk.com @theburningmonk Independent Consultant advisetraining delivery
- 17. API Gateway Lambda DynamoDB
- 18. API Gateway Lambda DynamoDBhow long did this req take?
- 19. what is the state of the world?
- 20. In control theory, observability is a measure of how well internal states of a system can be inferred from knowledge of its external outputs. what are the most important outputs to collect?
- 21. macro micro how well is this service performing in general? why did this transaction perform poorly?
- 22. API Gateway API GatewayLambda Lambda DynamoDB Service A Service B
- 23. API Gateway API GatewayLambda Lambda DynamoDB Service A Service B how long did service B took to respond?
- 24. API Gateway API GatewayLambda Lambda DynamoDB Service A Service B how long did service B took to respond? was DynamoDB slow? was it a cold start? could it be API Gateway?
- 25. API Gateway
- 26. Lambda
- 27. Lambda Duration
- 28. Lambda time to create and initialize the worker instance
- 31. Lambda time to create and initialize the worker instance
- 32. for API functions, use API Gateway’s IntegrationLatency as a proxy for “total response time from Lambda”
- 33. DynamoDB
- 35. “I'm facing this problem now with a lambda that usually takes 25 ms but once a week or so takes > 6000 ms and times out. The lambda's first step is to load a DynamoDB table that only has 8 items. I'm at a loss to understand how such a simple query could take so long.”
- 36. START
- 39. START 1st attempt exponential backoff (1) 2nd attempt exponential backoff (2)
- 40. START 1st attempt exponential backoff (1) 2nd attempt exponential backoff (2) 3rd attempt exponential backoff (3)
- 41. START 1st attempt exponential backoff (1) 2nd attempt exponential backoff (2) 3rd attempt exponential backoff (3) 4th attempt success!
- 42. START 1st attempt exponential backoff (1) 2nd attempt exponential backoff (2) 3rd attempt exponential backoff (3) 4th attempt success! SuccessfulRequestLatency
- 43. JavaScript AWS SDK 10 retries Initial exponential backoff of 50ms delay = Math.random() * (Math.pow(2, retryCount) * base) this is Marc Brooker’s fav formula!
- 44. 10 retries Initial exponential backoff of 50ms delay = Math.random() * (Math.pow(2, retryCount) * base) JavaScript AWS SDK
- 45. 10 retries Initial exponential backoff of 50ms delay = Math.random() * (Math.pow(2, retryCount) * base) JavaScript AWS SDK danger zone!
- 46. Record client-side latency metrics for IO operations
- 48. Embedded Metric Format (EMF)
- 51. API Gateway’s latency overhead IntegrationLatency [API Gateway] Latency [API Gateway]
- 52. Duration [Lambda] API Gateway’s latency overhead IntegrationLatency [API Gateway] Latency [API Gateway]
- 53. Duration [Lambda] Lambda’s allocation time API Gateway’s latency overhead IntegrationLatency [API Gateway] Latency [API Gateway]
- 54. SuccessfulRequestLatency [DynamoDB] Duration [Lambda] Lambda’s allocation time API Gateway’s latency overhead IntegrationLatency [API Gateway] Latency [API Gateway]
- 55. Caller-side DynamoDB latency [custom metric] SuccessfulRequestLatency [DynamoDB] Duration [Lambda] Lambda’s allocation time API Gateway’s latency overhead IntegrationLatency [API Gateway] Latency [API Gateway]
- 56. Caller-side retries (mostly) Caller-side DynamoDB latency [custom metric] SuccessfulRequestLatency [DynamoDB] Duration [Lambda] Lambda’s allocation time API Gateway’s latency overhead IntegrationLatency [API Gateway] Latency [API Gateway]
- 57. Latency (ms) Time Latency IntegrationLatency Duration Caller-side DynamoDB latency SuccessfulRequestLatency
- 58. Latency (ms) Time Latency IntegrationLatency Duration Caller-side DynamoDB latency SuccessfulRequestLatency
- 59. how well is this service performing in general? macro
- 60. why did this transaction perform poorly? micro
- 61. X-Ray
- 62. X-Ray
- 63. X-Ray can be encapsulated in custom modules
- 65. X-Ray doesn’t add latency can see “system” overhead (e.g. allocation time)
- 66. X-Ray doesn’t add latency can see “system” overhead (e.g. allocation time) built-in sampling
- 67. X-Ray doesn’t add latency can see “system” overhead (e.g. allocation time) built-in sampling X-Ray SDK adds significant overhead
- 68. X-Ray doesn’t add latency can see “system” overhead (e.g. allocation time) built-in sampling X-Ray SDK adds significant overhead doesn’t trace TCP traffic (RDS/Elasticache)
- 69. X-Ray doesn’t add latency can see “system” overhead (e.g. allocation time) built-in sampling X-Ray SDK adds significant overhead doesn’t trace TCP traffic (RDS/Elasticache) poor support for saync event sources (only SNS)
- 70. X-Ray doesn’t add latency can see “system” overhead (e.g. allocation time) built-in sampling X-Ray SDK adds significant overhead doesn’t trace TCP traffic (RDS/Elasticache) poor support for saync event sources (only SNS) doesn’t capture request & response data
- 71. X-Ray doesn’t add latency can see “system” overhead (e.g. allocation time) built-in sampling X-Ray SDK adds significant overhead doesn’t trace TCP traffic (RDS/Elasticache) poor support for saync event sources (only SNS) doesn’t capture request & response data logs and traces are separate
- 72. X-Ray doesn’t add latency can see “system” overhead (e.g. allocation time) built-in sampling X-Ray SDK adds significant overhead doesn’t trace TCP traffic (RDS/Elasticache) poor support for saync event sources (only SNS) doesn’t capture request & response data logs and traces are separate difficult to search
- 73. X-Ray good enough for simple workloads when you outgrow X-Ray, look for a 3rd-party tool
- 82. answer both macro and micro level questions in just a few clicks!
- 84. Support async event sources such as Kinesis, DynamoDB streams and SNS
- 85. Support TCP traffic - e.g. RDS and Elasticache
- 93. trace 500K invocations per month for FREE with promo code Yan500 platform.lumigo.io/signup
- 94. How to mitigate slow dependencies?
- 95. it depends…
- 96. can you use another service?
- 97. if not, a good caching strategy often helps
- 98. bit.ly/3h7Bo41
- 100. Client Server 1 Server 2 50ms later
- 103. runing required for each service
- 104. helps in some cases but can exaspate the problem in other cases
- 105. can you use another service?
- 107. platform.lumigo.io/signup trace 500K invocations per month for FREE with promo code Yan500