Serverless observability - a hero's perspective
•0 likes•385 views
Report
Share
Download to read offline
Yan Cui, an AWS Serverless Hero, will talk about the learnings from using serverless at scale. He will cover the challenges for observability in serverless asynchronous workloads and the patterns to address those challenges, like using centralized logging, correlation IDs, tracing, lambda extensions.
Recommended
More Related Content
What's hot
What's hot(20)
Similar to Serverless observability - a hero's perspective
Similar to Serverless observability - a hero's perspective(20)
More from Yan Cui
More from Yan Cui(20)
Recently uploaded
Recently uploaded(20)
Serverless observability - a hero's perspective
- 1. Serverless OBSERVABILITY A hero’s perspective
- 2. @theburningmonk theburningmonk.com What do I mean by “observability”?
- 3. Monitoring watching out for known failure modes in the system, e.g. network I/O, CPU, memory usage, …
- 4. @theburningmonk theburningmonk.com Observability being able to debug the system, and gain insights into the system’s behaviour
- 5. @theburningmonk theburningmonk.com In control theory, observability is a measure of how well internal states of a system can be inferred from knowledge of its external outputs. https://en.wikipedia.org/wiki/Observability
- 6. @theburningmonk theburningmonk.com In control theory, observability is a measure of how well internal states of a system can be inferred from knowledge of its external outputs. https://en.wikipedia.org/wiki/Observability including non- functional outputs
- 7. @theburningmonk theburningmonk.com These are the four pillars of the Observability Engineering team’s charter: • Monitoring • Alerting/Visualization • Distributed systems tracing infrastructure • Log aggregation/analytics “ ” http://bit.ly/2DnjyuW - Observability Engineering at Twitter
- 8. @theburningmonk theburningmonk.com microservices death stars circa 2015
- 9. @theburningmonk theburningmonk.com microservices death stars circa 2015 mm… I wonder what’s going on here…
- 13. @theburningmonk theburningmonk.com microservices death stars circa 2015 I got this!
- 16. Yan Cui http://theburningmonk.com @theburningmonk AWS user for 10 years
- 20. Yan Cui http://theburningmonk.com @theburningmonk Independent Consultant advise training delivery
- 22. Learn GraphQL and AppSync by building a Twitter clone with these technologies appsyncmasterclass.com
- 25. @theburningmonk theburningmonk.com Concurrency is in your application
- 26. @theburningmonk theburningmonk.com Concurrency is in your application Agent
- 27. @theburningmonk theburningmonk.com Concurrency is in your application Agent
- 28. @theburningmonk theburningmonk.com Concurrency is in your application Agent Shares the same CPU, memory and bandwidth as application
- 31. @theburningmonk theburningmonk.com API Gateway Lambda Worker Worker Worker Worker Worker Worker Worker Worker Worker
- 32. @theburningmonk theburningmonk.com API Gateway Lambda Worker Worker Worker Worker Worker Worker Worker Worker Worker Busy Busy Busy Busy Busy Busy
- 33. @theburningmonk theburningmonk.com API Gateway Lambda Worker Worker Worker Worker Worker Worker Worker Worker Worker Busy Busy Busy Busy Busy Busy
- 34. @theburningmonk theburningmonk.com API Gateway Lambda Worker Worker Worker Worker Worker Worker Worker Worker Worker Busy Busy Busy Busy Busy Busy Only processes one request at a time
- 35. @theburningmonk theburningmonk.com API Gateway Lambda Worker Worker Worker Worker Worker Worker Worker Worker Worker Busy Busy Busy Busy Busy Busy Only processes one request at a time Runs in an AWS-managed baremetal EC2 instance
- 36. @theburningmonk theburningmonk.com Resource overhead is irrelevant. Think latency overhead instead.
- 37. @theburningmonk theburningmonk.com Resource overhead is irrelevant. Think latency overhead instead.
- 39. @theburningmonk theburningmonk.com Built-in collection of: Metrics Logs Traces CloudWatch Metrics CloudWatch Logs X-Ray
- 40. challenges
- 42. @theburningmonk theburningmonk.com user request user request user request user request user request user request user request critical paths: minimise user-facing latency handler handler handler handler handler handler handler
- 43. @theburningmonk theburningmonk.com user request user request user request user request user request user request user request critical paths: minimise user-facing latency StatsD handler handler handler handler handler handler handler rsyslog background processing: batched, asynchronous, low overhead
- 44. @theburningmonk theburningmonk.com No background processing*.
- 45. @theburningmonk theburningmonk.com No background processing*. * you can work around this with Lambda extensions
- 46. (potential) Data Loss When workers are garbage collected
- 48. @theburningmonk theburningmonk.com Lambda data is batched between invocations
- 49. @theburningmonk theburningmonk.com Lambda idle data is batched between invocations
- 50. @theburningmonk theburningmonk.com Lambda idle garbage collection data is batched between invocations
- 52. @theburningmonk theburningmonk.com my code send metrics
- 53. @theburningmonk theburningmonk.com my code send metrics
- 54. @theburningmonk theburningmonk.com my code send metrics internet internet press button something happens
- 56. Lambda Extension to the rescue!
- 57. Lambda Extension to the rescue! Think “sidecar” but for Lambda ;-)
- 58. @theburningmonk theburningmonk.com Your code Time Invoke Idle Shutdown Runtime shutdown
- 59. @theburningmonk theburningmonk.com Your code Time Extension Invoke Idle Runtime shutdown Shutdown
- 60. @theburningmonk theburningmonk.com Your code Time Extension Invoke Idle Runtime shutdown Shutdown
- 61. @theburningmonk theburningmonk.com Your code Time Extension Invoke Idle Shutdown 2s Extension shutdown Runtime shutdown
- 62. @theburningmonk theburningmonk.com Lambda idle garbage collection data is batched between invocations
- 63. @theburningmonk theburningmonk.com Your code Time Extension Invoke Idle Shutdown 2s Extension shutdown Runtime shutdown Flush buffered telemetry data
- 64. @theburningmonk theburningmonk.com Your code Time Extension Invoke Idle Shutdown 2s Extension shutdown Runtime shutdown 5-10 mins on average
- 65. @theburningmonk theburningmonk.com my code send metrics
- 66. @theburningmonk theburningmonk.com Your code Time Extension Invoke Idle Shutdown 2s Extension shutdown Runtime shutdown
- 67. @theburningmonk theburningmonk.com Your code Time Extension Invoke Idle Shutdown 2s Extension shutdown Runtime shutdown You pay for this time too
- 68. @theburningmonk theburningmonk.com Your code Time Extension Invoke Idle Shutdown 2s Extension shutdown Runtime shutdown Return function response
- 69. @theburningmonk theburningmonk.com Your code Time Extension Invoke Idle Shutdown 2s Extension shutdown Runtime shutdown Return function response Send telemetry data
- 70. Lots of async invocations
- 72. @theburningmonk theburningmonk.com ? functions are often chained together via asynchronous invocations
- 73. @theburningmonk theburningmonk.com ? SNS Kinesis EventBridge CloudWatch Logs IoT DynamoDB S3 SES
- 75. @theburningmonk theburningmonk.com S3 Lambda DynamoDB Lambda
- 76. @theburningmonk theburningmonk.com S3 Lambda DynamoDB Lambda API Gateway Lambda
- 77. @theburningmonk theburningmonk.com S3 Lambda DynamoDB Lambda API Gateway Lambda EventBridge Lambda
- 78. @theburningmonk theburningmonk.com S3 Lambda DynamoDB Lambda API Gateway Lambda EventBridge Lambda Lambda
- 79. @theburningmonk theburningmonk.com Tracing through asynchronous invocations can be difficult
- 80. @theburningmonk theburningmonk.com Use correlation IDs to find related logs
- 81. @theburningmonk theburningmonk.com S3 Lambda DynamoDB Lambda API Gateway Lambda EventBridge Lambda Lambda How to inject correlation IDs?
- 82. @theburningmonk theburningmonk.com S3 Lambda DynamoDB Lambda API Gateway Lambda EventBridge Lambda Lambda How to inject correlation IDs? How to inject correlation IDs? How to inject correlation IDs?
- 84. @theburningmonk theburningmonk.com S3 Lambda DynamoDB Lambda API Gateway Lambda EventBridge Lambda Lambda
- 85. @theburningmonk theburningmonk.com S3 Lambda DynamoDB Lambda API Gateway Lambda EventBridge Lambda Lambda Microservice 1 Microservice 2 Microservice 3
- 86. @theburningmonk theburningmonk.com S3 Lambda DynamoDB Lambda API Gateway Lambda EventBridge Lambda Lambda Microservice 1 Microservice 2 Microservice 3 AWS account 1 AWS account 2 AWS account 3
- 87. Log Aggregation
- 91. @theburningmonk theburningmonk.com CloudWatch Logs is an async event source for Lambda
- 92. @theburningmonk theburningmonk.com Concurrent Executions Time regional max concurrency functions that are delivering business value
- 93. @theburningmonk theburningmonk.com Concurrent Executions Time regional max concurrency functions that are delivering business value ship logs
- 94. @theburningmonk theburningmonk.com either set concurrency limit on the log shipping function (and potentially lose logs due to throttling) or…
- 96. @theburningmonk theburningmonk.com 1 shard = 1 concurrent execution i.e. control the no. of concurrent executions with no. of shards
- 97. @theburningmonk theburningmonk.com or consider using Lambda Extensions
- 98. @theburningmonk theburningmonk.com or consider using Lambda Extensions Optionally disable CloudWatch Logs to save on cost
- 99. @theburningmonk theburningmonk.com Your code Time Extension Invoke Idle Shutdown 2s Extension shutdown Runtime shutdown Return function response Send logs
- 100. Distributed Tracing
- 101. @theburningmonk theburningmonk.com S3 Lambda DynamoDB Lambda API Gateway Lambda EventBridge Lambda Lambda
- 102. X-Ray
- 105. @theburningmonk theburningmonk.com Some services are not supported e.g. Kinesis, DynamoDB
- 107. @theburningmonk theburningmonk.com Requires manual instructation of AWS SDK and HTTP module
- 108. AWS offers a lot of built-in tools, but…
- 109. @theburningmonk theburningmonk.com assembly are required
- 110. @theburningmonk theburningmonk.com assembly are required lots of decisions for you to make
- 111. @theburningmonk theburningmonk.com assembly are required
- 112. @theburningmonk theburningmonk.com Have to use (and learn) multiple services
- 113. @theburningmonk theburningmonk.com Not one complete package
- 114. Many vendors cater for serverless workloads, but…
- 115. @theburningmonk theburningmonk.com Their focus is still on serverful applications, and has shallow support for serverless architectures (e.g. poor support for async workloads)