Download as PDF, PPTX
Uploaded byYan Cui
Beware the potholes on the road to serverless
The document discusses the concept of 'serverless' computing, emphasizing its advantages such as not needing to provision servers and only paying for usage. It highlights common pitfalls in serverless architecture, including failure to let go of legacy practices and the importance of using deployment toolkits. Additionally, it stresses the need for better observability and monitoring to optimize serverless applications.
More Related Content
Similar to Beware the potholes on the road to serverless
Beware the potholes on the road to serverless
- 1.
- 2. What do youmean by ‘serverless’?
- 3.
- 5. Gojko Adzic It isserverless the same way WiFi is wireless. http://bit.ly/2yQgwwb
- 6. Serverless means… don’t payfor it if no-one uses it don’t need to worry about scaling don’t need to provision and manage servers
- 7. in other words,it’s a lot like taking a cab
- 8. Ownership Fuel Navigate To get there! Focuson getting there!
- 9. HW Ownership OS Runtime &Scale Code Focus on getting there! Physical Servers Virtual Machines Containers Serverless
- 10. Nano Services SelfManaged Cost Paradigm ChangeAsync Dynamic agile env
- 11. “why are wefailing at this?”
- 12.
- 13.
- 14.
- 15.
- 16.
- 17. monolith serverless missing learnings frommicroservices
- 18. monolith serverless missing learnings frommicroservices poor decisions
- 19.
- 20.
- 21.
- 23.
- 24. https://theburningmonk.com/workshops Amsterdam, March 19-20Helsinki, May 4-5 Stockholm, May 14-15 Dublin, June 16-17 London, September 24-25 early bird until Feb 11
- 25. #1 not letting goof legacy thinking
- 26. “we’re doing serverless, butwhy aren’t thing going faster?”
- 27.
- 28. there are nosilver bullets
- 30. centralised team Team ATeam B Team C Team D …
- 31. “but the developersdon’t understand AWS and how our infrastructure is set up”
- 32. “but the developersdon’t understand AWS and how our infrastructure is set up” let’s solve this problem instead!
- 33. what got youhere won’t get you there
- 34. if (path ==“/user” && method == “GET”) { return getUser(…); } else if (path == “/user” && method == “DELETE”) { return deleteUser(…); } else if (path == “/user” && method == “POST”) { return createUser(…); } else if …. Monolithic Functions
- 35. GET /user POST /user DELETE/user Single-Purposed Functions
- 36. author: yan.cui feature: user-api user-api-dev MonolithicSingle-Purposed author: yan.cui feature: user-api user-api-dev-get-user author: yan.cui feature: user-api user-api-dev-create-user author: yan.cui feature: user-api user-api-dev-delete-user
- 37. author: yan.cui feature: user-api user-api-dev MonolithicSingle-Purposed author: yan.cui feature: user-api user-api-dev-get-user author: yan.cui feature: user-api user-api-dev-create-user author: yan.cui feature: user-api user-api-dev-delete-user find related functions by prefix
- 38. author: yan.cui feature: user-api user-api-dev MonolithicSingle-Purposed author: yan.cui feature: user-api user-api-dev-get-user author: yan.cui feature: user-api user-api-dev-create-user author: yan.cui feature: user-api user-api-dev-delete-user discoverability (without having to dig into the code)
- 39. author: yan.cui feature: user-api user-api-dev MonolithicSingle-Purposed author: yan.cui feature: user-api user-api-dev-get-user author: yan.cui feature: user-api user-api-dev-create-user author: yan.cui feature: user-api user-api-dev-delete-user what does it do?
- 40. author: yan.cui feature: user-api user-api-dev MonolithicSingle-Purposed author: yan.cui feature: user-api user-api-dev-get-user author: yan.cui feature: user-api user-api-dev-create-user author: yan.cui feature: user-api user-api-dev-delete-user dynamodb:GetItem dynamodb:PutItem dynamodb:DeleteItem
- 41. author: yan.cui feature: user-api user-api-dev MonolithicSingle-Purposed author: yan.cui feature: user-api user-api-dev-get-user author: yan.cui feature: user-api user-api-dev-create-user author: yan.cui feature: user-api user-api-dev-delete-user dynamodb:GetItem dynamodb:PutItem dynamodb:DeleteItem no least privilege…
- 42. author: yan.cui feature: user-api user-api-dev MonolithicSingle-Purposed author: yan.cui feature: user-api user-api-dev-get-user author: yan.cui feature: user-api user-api-dev-create-user author: yan.cui feature: user-api user-api-dev-delete-user require(x) require(y) require(z)
- 43. author: yan.cui feature: user-api user-api-dev MonolithicSingle-Purposed author: yan.cui feature: user-api user-api-dev-get-user author: yan.cui feature: user-api user-api-dev-create-user author: yan.cui feature: user-api user-api-dev-delete-user require(x) require(y) require(z)
- 45. more dependecies equals slowercold start
- 46. author: yan.cui feature: user-api user-api-dev MonolithicSingle-Purposed author: yan.cui feature: user-api user-api-dev-get-user author: yan.cui feature: user-api user-api-dev-create-user author: yan.cui feature: user-api user-api-dev-delete-user require(x) require(y) require(z) worse cold start performance
- 47. keep functions simple,and single-purposed
- 48. #2 one account thatrules them all
- 49. mind the sharedlimits
- 50. no. of DynamoDBtables no. of API Gateway regional APIs no. of API Gateway edge-optimized APIs no. of Kinesis shards no. of IAM roles no. of S3 buckets no. of CloudFormation stacks no. of SNS subscription filters no. of SSM parameters … Resource Limits
- 51. DynamoDB read &write API Gateway requests/second Lambda concurrent executions SSM parameter ops/second … Throughput Limits
- 53.
- 54. One account perTeam per Environment
- 56.
- 57.
- 61. the platforms needto do better at educating users on how to choose between different services
- 62. SNS vs SQSvs Kinesis vs MKS? the platforms need to do better at educating users on how to choose between different services
- 63. ordering replay events Kinesis SQSSNS by shard none (standard) global (FIFO) none up to 7 days none none mode retry batched batched (up to 10) singular retried until success (customizable) retry + DLQ retry + DLQ concurrency 1 per shard auto-scaled fan-out!!! subscribers many one-to-one many EventBridge many none none singular retry + DLQ fan-out!!!
- 64.
- 65.
- 66. #4 not using adeployment toolkit
- 68.
- 69. don’t write yourown deployment framework
- 70.
- 73. #6 one repo perfunction
- 74.
- 75.
- 76.
- 77.
- 78. one repo perservice?
- 79.
- 80.
- 88.
- 89. CI/CD pipeline perservice
- 90. functions are deployedtogether, as a stack
- 91. unencrypted secrets in envvars #7
- 92. secrets should NEVERbe in plain text in env variables
- 93. SSM Parameter Store Secret1 Secret 2 IAM Environment: SECRET_1: … SECRET_2: … Environment: SECRET_1: … SECRET_2: …
- 94. SSM Parameter Store Secret1 Secret 2 IAM Environment: SECRET_1: … SECRET_2: … Environment: SECRET_1: … SECRET_2: … yay!
- 97. SSM Parameter Store Secret1 Secret 2 IAM fetch at cold start, cache, invalidate every x mins
- 98.
- 100. SSM Parameter Store Secret1 Secret 2 IAM switch to Higher Throughput if you need more than 40 ops/s
- 101. not following least privilegeprinciple #8
- 105.
- 106. async sync S3 SNS SES CloudFormation CloudWatch Logs CloudWatchEvents Scheduled Events CodeCommit AWS Config http://amzn.to/2v7Kc3b Cognito Alexa Lex API Gateway pulling DynamoDB Stream Kinesis Stream SQS
- 107. async sync S3 SNS SES CloudFormation CloudWatch Logs CloudWatchEvents Scheduled Events CodeCommit AWS Config http://amzn.to/2vs2lIg Cognito Alexa Lex API Gateway pulling DynamoDB Stream Kinesis Stream SQS Lambda handles retries (twice, then DLQ)
- 108. configure DLQ forasync functions so you don’t lose failed events
- 111. DLQ Lambda Destinations payloadpayload, context(s), and response
- 113. DLQ Lambda Destinations payloadpayload, context(s), and response DeadLetterErrors no error metrics!
- 114.
- 115. “Lambda generates toomuch load for the downstream system”
- 116.
- 117.
- 118. ordering replay events Kinesis SQSSNS by shard none (standard) global (FIFO) none up to 7 days none none mode retry batched batched (up to 10) singular retried until success (customizable) retry + DLQ retry + DLQ concurrency 1 per shard auto-scaled fan-out!!! subscribers many one-to-one many EventBridge many none none singular retry + DLQ fan-out!!!
- 119. if you want… maximum throughput SNS precisecontrol over throughput Kinesis
- 120. if you want… maximum throughput SNS precisecontrol over throughput Kinesis how quickly it scales out
- 121. if you want… maximum throughput SNS precisecontrol over throughput Kinesis how quickly it scales out SQS DynamoDB Streams
- 122. ordering replay events Kinesis SQSSNS by shard none (standard) global (FIFO) none up to 7 days none none mode retry batched batched (up to 10) singular retried until success (customizable) retry + DLQ retry + DLQ concurrency 1 per shard auto-scaled fan-out!!! subscribers many one-to-one many EventBridge many none none singular retry + DLQ fan-out!!!
- 123.
- 124. “cold starts onlyhappen to the first request”
- 125. function invocationconcurrent execution i.e.a container
- 126. function invocationconcurrent execution i.e.a container class instance method call
- 127. Lambda scales thenumber of concurrent executions based on traffic
- 128. existing “containers” arereused where possible
- 129.
- 130.
- 131.
- 132.
- 133.
- 134.
- 135.
- 136.
- 137.
- 138.
- 143.
- 144. Lambda warmers don’twork when you have > 1 concurrent executions
- 145.
- 146. FREQUENCY DURATION dictated byuser traffic, out of your control
- 147. cold starts isgenerally not an issue if you have a steady traffic pattern
- 148.
- 149.
- 150.
- 151.
- 152. minimise the durationof cold starts so they fall within acceptable latency range
- 153.
- 154.
- 155.
- 156. there are nosilver bullets
- 157. reserved concurrency isa powerful tool IFF you have a cold start problem don’t use it by default
- 158.
- 159. default RDS configsare bad for Lambda
- 160. default RDS configsare bad for Lambda idle connections are not closed too many connections per “container” max open connection is too low
- 161.
- 162. set “wait_timeout” and“interactive_timeout” to 10 mins (default is 8 hours!)
- 163.
- 164. set client socketpool size to 1
- 167.
- 168. happened system repaireduserimpact reduce MTTR
- 169.
- 170.
- 171. happened system repaireduserimpact MTTDiscovery
- 173. “What alerts shouldI have?”
- 174. It depends onwhat you’re building…
- 175. But, this isa good starting point
- 176. Lambda error rate % throttlecount DLR error count iterator age regional concurrency
- 177. Lambda error rate % throttlecount DLR error count iterator age regional concurrency API Gateway p90/95/99 latency success rate % 4xx rate % 5xx rate %
- 178. API Gateway p90/95/99 latency successrate % 4xx rate % 5xx rate % SQS message age Lambda error rate % throttle count DLR error count iterator age regional concurrency
- 179. API Gateway p90/95/99 latency successrate % 4xx rate % 5xx rate % SQS message age Step Functions failed count throttle count timed out count Lambda error rate % throttle count DLR error count iterator age regional concurrency
- 180. SQS message age Step Functions failedcount throttle count timed out count API Gateway p90/95/99 latency success rate % 4xx rate % 5xx rate % Lambda error rate % throttle count DLR error count iterator age regional concurrency
- 181. monitor and alerton message flow rate for event processing pipelines
- 182. “Can’t you codifythese?”
- 185. https://theburningmonk.com/hire-me AdviseTraining Delivery “Fundamentally, Yanhas improved our team by increasing our ability to derive value from AWS and Lambda in particular.” Nick Blair Tech Lead
- 186.
- 187. bit.ly/prod-ready-sls-ams-2019 Backspace Escape Rooms Transformatorweg30A 20% off with aws-ug-ams-0109
- 188. https://theburningmonk.com/workshops Amsterdam, March 19-20Helsinki, May 4-5 Stockholm, May 14-15 Dublin, June 16-17 London, September 24-25 early bird until Feb 11
- 189.