Download as PDF, PPTX
Uploaded byYan Cui
Apply best parts of microservices to serverless
The document discusses the integration of microservices principles into serverless architectures, emphasizing observability and the importance of minimizing latency. It outlines key lessons for designing resilient systems, such as avoiding shared databases, managing spiky loads, and preparing for inevitable failures. The author also highlights best practices for logging and monitoring to improve performance and incident resolution in a serverless environment.
More Related Content
Similar to Apply best parts of microservices to serverless
Apply best parts of microservices to serverless
- 1. applying the bestparts of Microservices to Serverless
- 2.
- 4. follow @DAZN_ngnrs for updatesabout the engineering team We’re hiring! Visit engineering.dazn.com to learn more.
- 6.
- 7.
- 8.
- 9.
- 10.
- 11. SQL NoSQL OOP Functional OnPremise Cloud Waterfall Agile Monoliths Microservices
- 12.
- 13.
- 14.
- 17.
- 18. what’s this? this solvesall my problems!
- 19. what’s this? this solvesall my problems! this is rubbish!
- 20. what’s this? this solvesall my problems! this is rubbish! I’m starting to get it..
- 21. what’s this? this solvesall my problems! this is rubbish! I’m starting to get it.. I know what I’m doing
- 22. SQL NoSQL OOP Functional OnPremise Cloud Waterfall Agile Monoliths Microservices Server-ful Serverless
- 23. “those who cannotremember the past are condemned to repeat it” - George Santayana
- 24. what’s this? this solvesall my problems! this is rubbish! I’m starting to get it.. I know what I’m doing
- 25. lesson 1. don’tfly blind
- 26.
- 27.
- 28.
- 30. These are thefour pillars of the Observability Engineering team’s charter: • Monitoring • Alerting/visualization • Distributed systems tracing infrastructure • Log aggregation/analytics “ ” http://bit.ly/2DnjyuW- Observability Engineering at Twitter
- 31.
- 32.
- 33. user request user request userrequest user request user request user request user request critical paths: minimise user-facing latency handler handler handler handler handler handler handler
- 34. user request user request userrequest user request user request user request user request critical paths: minimise user-facing latency StatsD handler handler handler handler handler handler handler rsyslog background processing: batched, asynchronous, low overhead
- 35. user request user request userrequest user request user request user request user request critical paths: minimise user-facing latency StatsD handler handler handler handler handler handler handler rsyslog background processing: batched, asynchronous, low overhead NO background processing except what platform provides
- 37. 2016-07-12T12:24:37.571Z 994f18f9-482b-11e6-8668-53e4eab441ae GOT isoff air, what do I do now?
- 38. 2016-07-12T12:24:37.571Z 994f18f9-482b-11e6-8668-53e4eab441ae GOT isoff air, what do I do now? UTC Timestamp Request Id your log message
- 40. one log groupper function one log stream for each concurrent invocation
- 41. logs are noteasily searchable in CloudWatch Logs me
- 42.
- 43. CloudWatch Logs AWSLambda ELK stack
- 44.
- 45. you need touse structured logging me
- 47. CloudWatch Logs $0.50 perGB ingested $0.03 per GB archived per month
- 48. CloudWatch Logs $0.50 perGB ingested $0.03 per GB archived per month 1M invocation of a 128MB function = $0.000000208 * 1M + $0.20 = $0.408
- 49. DON’T leave debuglogging ON in production
- 50. you need tosample debug logs in production me
- 51. volume of logs observability alldebug logs no debug logs sampled debug logs hurts mean time to resolution (MTTR) during a production incident
- 52. volume of logs observability alldebug logs no debug logs sampled debug logs $$$$$$
- 53. always log theinvocation event on error me
- 54.
- 60.
- 62.
- 63.
- 64. my code send metrics internetinternet press button something happens
- 65. those extra 10-20msfor sending custom metrics would compound when you have microservices and multiple APIs are called within one slice of user event
- 66. Amazon found every100ms of latency cost them 1% in sales. http://bit.ly/2EXPfbA
- 67. console.log(“hydrating yubls fromdb…”); console.log(“fetching user info from user-api”); console.log(“MONITORING|1489795335|27.4|latency|user-api-latency”); console.log(“MONITORING|1489795335|8|count|yubls-served”); timestamp metric value metric type metric namemetrics logs
- 68. CloudWatch Logs AWSLambda ELK stack logs m etrics CloudWatch
- 70. API Gateway send custommetrics asynchronously
- 71. SNS KinesisS3API Gateway … sendcustom metrics asynchronously send custom metrics as part of function invocation
- 73.
- 74. ? functions are oftenchained together via asynchronous invocations
- 75.
- 76. ? SNS Kinesis CloudWatch Events CloudWatch LogsIoT DynamoDB S3 SES tracing ASYNCHRONOUS invocationsthrough so many different event sources is difficult
- 77.
- 79. do not spanover API Gateway
- 80. narrow focus ona function good for homing in on performance issues for a particular function, but offers little to help you build intuition about how your system operates as a whole.
- 81. don’t span overasync invocations good for identifying dependencies of a function, but not good enough for tracing the entire call chain as user request/data flows through the system via async event sources.
- 82. don’t span overnon-AWS services
- 83.
- 85.
- 87. correlation IDs* * eg.request-id, user-id, yubl-id, etc.
- 88.
- 89.
- 90. lesson 2. noshared DBs
- 91. shared DBs createTIGHT COUPLING between services
- 92. build loosely-coupled systemthrough events
- 93. service A serviceB service C service D bounded context bounded context
- 94. service A serviceB service C service D bounded context bounded context
- 95. service A serviceB service C service D bounded context bounded context
- 96. service A serviceB service C service D bounded context bounded context
- 97. lesson 3. spikyload between services
- 98.
- 99. downstream systems mightnot be as scalable
- 101. service A serviceB Kinesis Lambda
- 102. service A serviceB Kinesis Lambda concurrency == no. of shards
- 104. service A serviceB Kinesis Lambda retried until success
- 106. lesson 4. failuresare inevitable
- 107. complex distributed systemsfail in.. well, complex, sometimes cascaded ways..
- 108. the only wayto truly know your system’s resilience against failures is to test it through controlled experiments
- 109. there are moreinherent chaos and complexity in a Serverless architecture
- 110. smaller units ofdeployment but A LOT more of them!
- 111. more difficult toharden around boundaries serverful serverless
- 112.
- 113. ? Kinesis CloudWatch Events CloudWatch LogsIoT DynamoDB S3 more intermediary services, andgreater variety too SNS SES
- 114. ? Kinesis CloudWatch Events CloudWatch LogsIoT DynamoDB S3 more intermediary services, andgreater variety too each with its own set of failure modes SNS SES
- 115. more configurations, more opportunitiesfor misconfiguration serverful serverless
- 116. more unknown failuremodes in infrastructure that we don’t control
- 117. often there’s littlewe can do when an outage occurs in the platform
- 118.
- 119.
- 120. missing fallback whendownstream is unavailable
- 121.
- 123.
- 124. inject failures validate failurehandling
- 126.
- 127.
- 128. “those who cannotremember the past are condemned to repeat it” - George Santayana
- 129.
- 130.
- 131. amortize spiky loadbetween services
- 132.
- 133.