Download as PDF, PPTX
![get-restaurants
global.CONTEXT
global.CONTEXT
x-correlation-id = …
x-correlation-xxx = …
get-index
headers[“User-Agent”]
headers[“Debug-Log-Enabled”]
headers[“User-Agent”]
headers[“Debug-Log-Enabled”]
headers[“x-correlation-id”]
capture
forward
function
event
log.info(…)](https://image.slidesharecdn.com/serverlessobservability-181115192416/85/How-to-build-observability-into-Serverless-BuildStuff-2018-141-320.jpg)
Uploaded byYan Cui
How to build observability into Serverless (BuildStuff 2018)
The document discusses the challenges and strategies for building observability into serverless architectures, emphasizing the importance of monitoring, alerting, and tracing in understanding system behavior. It highlights the concept of 'survivor bias' in monitoring techniques, where only known failure modes are addressed, and the need for structured logging and correlation IDs across microservices. Additionally, it outlines the four pillars of observability engineering, focusing on log aggregation, analytics, and minimizing user-facing latency.
More Related Content
Similar to How to build observability into Serverless (BuildStuff 2018)
![Getting Started with Apache Spark: Big Data Made Simple [Free Meetup]](https://cdn.slidesharecdn.com/ss_thumbnails/apachesparkgettingstarted-260203175547-8361bcc3-thumbnail.jpg?width=640&height=640&fit=bounds)
How to build observability into Serverless (BuildStuff 2018)
- 1. How to buildobservability into Serverless Yan Cui @theburningmonk
- 3.
- 4.
- 5.
- 6. Abraham Wald Wald notedthat the study only considered the aircraft that had survived their missions—the bombers that had been shot down were not present for the damage assessment. The holes in the returning aircraft, then, represented areas where a bomber could take damage and still return home safely.
- 7. Abraham Wald Wald notedthat the study only considered the aircraft that had survived their missions—the bombers that had been shot down were not present for the damage assessment. The holes in the returning aircraft, then, represented areas where a bomber could take damage and still return home safely.
- 8. survivor bias inmonitoring
- 9. survivor bias inmonitoring Only focus on failure modes that we were able to successfully identify through investigation and postmortem in the past. The bullet holes that shot us down and we couldn’t identify stay invisible, and will continue to shoot us down.
- 10. What do Imean by “observability”?
- 11. Monitoring watching out for knownfailure modes in the system, e.g. network I/O, CPU, memory usage, …
- 12. Observability being able todebug the system, and gain insights into the system’s behaviour
- 13. In control theory,observability is a measure of how well internal states of a system can be inferred from knowledge of its external outputs. https://en.wikipedia.org/wiki/Observability
- 14.
- 15.
- 16. Known SuccessKnown Errors easyto monitor!
- 17. Known SuccessKnown Errors KnownUnknowns
- 18. Known SuccessKnown Errors KnownUnknownsUnknown Unknowns
- 19. Known SuccessKnown Errors KnownUnknownsUnknown Unknowns invisible bullet holes
- 20. Known SuccessKnown Errors KnownUnknownsUnknown Unknowns
- 21. Known SuccessKnown Errors KnownUnknownsUnknown Unknowns only alert on this
- 22. Known SuccessKnown Errors KnownUnknownsUnknown Unknowns alert on the absence of this!
- 23. Known SuccessKnown Errors KnownUnknownsUnknown Unknowns what went wrong?
- 24. These are thefour pillars of the Observability Engineering team’s charter: • Monitoring • Alerting/Visualization • Distributed systems tracing infrastructure • Log aggregation/analytics “ ” http://bit.ly/2DnjyuW- Observability Engineering at Twitter
- 25. microservices death starscirca 2015
- 26. mm… I wonderwhat’s going on here… microservices death stars circa 2015
- 31. I got this! microservicesdeath stars circa 2015
- 34. About me ▪ PrincipalEngineer at DAZN ▪ AWS Serverless Hero ▪ Author of Production-Ready Serverless* by Manning ▪ Blogger** ▪ Speaker * https://bit.ly/production-ready-serverless ** https://theburningmonk.com
- 37.
- 38.
- 39. About DAZN ▪ Availablein 7 countries - Austria, Switzerland, Germany, Japan, Canada, Italy and USA ▪ Available on 30+ platforms
- 40. About DAZN ▪~1,000,000 concurrentviewers at peak
- 41. follow @dazneng for updatesabout the engineering team We’re hiring! Visit engineering.dazn.com to learn more. WE’RE HIRING!
- 42.
- 43.
- 44.
- 45. •nowhere to installagents/daemons new challenges
- 46. user request user request userrequest user request user request user request user request critical paths: minimise user-facing latency handler handler handler handler handler handler handler
- 47. user request user request userrequest user request user request user request user request critical paths: minimise user-facing latency StatsD handler handler handler handler handler handler handler rsyslog background processing: batched, asynchronous, low overhead
- 48. user request user request userrequest user request user request user request user request critical paths: minimise user-facing latency StatsD handler handler handler handler handler handler handler rsyslog background processing: batched, asynchronous, low overhead NO background processing except what platform provides
- 49. •no background processing •nowhereto install agents/daemons new challenges
- 50. EC2 concurrency used tobe handled by your code
- 51. EC2 Lambda Lambda Lambda Lambda Lambda now, it’s handledby the AWS Lambda platform
- 52. EC2 logs & metricsused to be batched here
- 53. EC2 Lambda Lambda Lambda Lambda Lambda now, they arebatched in each concurrent execution, at best…
- 56. HIGHER concurrency tolog aggregation/telemetry system
- 57. •higher concurrency totelemetry system •nowhere to install agents/daemons •no background processing new challenges
- 58.
- 59. Lambda data is batchedbetween invocations
- 60. Lambda idle data is batchedbetween invocations
- 61. Lambda idle garbage collectiondata isbatched between invocations
- 62. Lambda idle garbage collectiondata isbatched between invocations HIGH chance of data loss
- 63. •high chance ofdata loss (if batching) •nowhere to install agents/daemons •no background processing •higher concurrency to telemetry system new challenges
- 64.
- 65.
- 66.
- 67. my code send metrics internetinternet press button something happens
- 69.
- 70. ? functions are oftenchained together via asynchronous invocations
- 71.
- 72. ? SNS Kinesis CloudWatch Events CloudWatch LogsIoT DynamoDB S3 SES tracing ASYNCHRONOUS invocationsthrough so many different event sources is difficult
- 73. •asynchronous invocations •nowhere toinstall agents/daemons •no background processing •higher concurrency to telemetry system •high chance of data loss (if batching) new challenges
- 74. These are thefour pillars of the Observability Engineering team’s charter: • Monitoring • Alerting/Visualization • Distributed systems tracing infrastructure • Log aggregation/analytics “ ” http://bit.ly/2DnjyuW- Observability Engineering at Twitter
- 75.
- 77. 2016-07-12T12:24:37.571Z 994f18f9-482b-11e6-8668-53e4eab441ae GOT isoff air, what do I do now?
- 78. 2016-07-12T12:24:37.571Z 994f18f9-482b-11e6-8668-53e4eab441ae GOT isoff air, what do I do now? UTC Timestamp Request Id your log message
- 80. one log groupper function one log stream for each concurrent invocation
- 81. logs are noteasily searchable in CloudWatch Logs me
- 82.
- 84. CloudWatch Logs isan async event source for Lambda
- 85. Concurrent Executions Time regional max concurrency functionsthat are delivering business value
- 86. Concurrent Executions Time regional max concurrency functionsthat are delivering business value ship logs
- 87. either set concurrencylimit on the log shipping function (and potentially lose logs due to throttling) or…
- 89. 1 shard =1 concurrent execution i.e. control the no. of concurrent executions with no. of shards
- 90.
- 91.
- 96.
- 97. use structured loggingwith JSON
- 98.
- 99.
- 101. traditional loggers aretoo heavy for Lambda
- 105. CloudWatch Logs $0.50 perGB ingested $0.03 per GB archived per month
- 106. CloudWatch Logs $0.50 perGB ingested $0.03 per GB archived per month 1M invocation of a 128MB function = $0.000000208 * 1M + $0.20 = $0.408
- 107. DON’T leave debuglogging ON in production
- 109. have to redeployALL the functions along the call path to collect all relevant debug logs
- 112.
- 114. EC2 Lambda Lambda Lambda Lambda Lambda Concurrency is handledby the AWS Lambda platform
- 120. sampling decision hasto be followed by an entire call chain
- 121. Initial Request ID UserID Session ID User-Agent Order ID …
- 122. every function needsto do the right thing and propagate information such as correlation IDs along to APIs, streams, queues, etc.
- 123. invest in toolsto make it easy to do the “right thing”
- 124.
- 125.
- 126.
- 127.
- 128. EC2 Lambda Lambda Lambda Lambda Lambda Concurrency is handledby the AWS Lambda platform
- 129. store correlation IDsin global variable
- 132. use middleware toauto-capture incoming correlation IDs
- 133. extract correlation IDsfrom invocation event, and store them in the correlation-ids module reset
- 135. logger to alwaysinclude captured correlation IDs
- 136. HTTP and AWSSDK clients to auto-forward correlation IDs on
- 137.
- 138.
- 139.
- 140. { “body”: null, “resource”: “/restaurants”, “headers”:{ “x-correlation-id”: “…” }, … } get-index get-restaurants
- 141. get-restaurants global.CONTEXT global.CONTEXT x-correlation-id = … x-correlation-xxx= … get-index headers[“User-Agent”] headers[“Debug-Log-Enabled”] headers[“User-Agent”] headers[“Debug-Log-Enabled”] headers[“x-correlation-id”] capture forward function event log.info(…)
- 144.
- 145.
- 147. •no background processing •nowhereto install agents/daemons new challenges
- 148. my code send metrics internetinternet press button something happens
- 149. those extra 10-20msfor sending custom metrics would compound when you have microservices and multiple APIs are called within one slice of user event
- 150. Amazon found every100ms of latency cost them 1% in sales. http://bit.ly/2EXPfbA
- 151. console.log(“hydrating yubls fromdb…”); console.log(“fetching user info from user-api”); console.log(“MONITORING|1489795335|27.4|latency|user-api-latency”); console.log(“MONITORING|1489795335|8|count|yubls-served”); timestamp metric value metric type metric namemetrics logs
- 152. CloudWatch Logs AWSLambda ELK stack logs m etrics CloudWatch
- 154.
- 155.
- 156. API Gateway send custommetrics asynchronously
- 157. SNS KinesisS3API Gateway … sendcustom metrics asynchronously send custom metrics as part of function invocation
- 160.
- 161.
- 165. don’t span overasync invocations good for identifying dependencies of a function, but not good enough for tracing the entire call chain as user request/data flows through the system via async event sources.
- 166. don’t span overnon-AWS services
- 170.
- 171.
- 172. make it easyto do the right thing
- 173. API Gateway andKinesis Authentication & authorisation (IAM, Cognito) Testing Running & Debugging functions locally Log aggregation Monitoring & Alerting X-Ray Correlation IDs CI/CD Performance and Cost optimisation Error Handling Configuration management VPC Security Leading practices (API Gateway, Kinesis, Lambda) Canary deployments http://bit.ly/prod-ready-serverless get 40% off with: ytcui
- 174. @theburningmonk theburningmonk.com github.com/theburningmonk API Gateway andKinesis Authentication & authorisation (IAM, Cognito) Testing Running & Debugging functions locally Log aggregation Monitoring & Alerting X-Ray Correlation IDs CI/CD Performance and Cost optimisation Error Handling Configuration management VPC Security Leading practices (API Gateway, Kinesis, Lambda) Canary deployments http://bit.ly/prod-ready-serverless get 40% off with: ytcui