Common mistakes in serverless adoption

1. WEBINARS

2. is the biggest tech conference for developers in EMEA, open to all languages and technologies.

3. SPECIAL DISCOUNT May 27-28, 2020 LET’S MEET IN CODEMOTION AMSTERDAM 2020! 20% DISCOUNT FOR YOU CODE: CodemotionAmsterdam20Cui

4. What do you mean by ‘serverless’?

5. @theburningmonk theburningmonk.com “Serverless”

6. @theburningmonk theburningmonk.com

7. @theburningmonk theburningmonk.com Gojko Adzic It is serverless the same way WiFi is wireless. http://bit.ly/2yQgwwb

8. @theburningmonk theburningmonk.com Serverless means… don’t pay for it if no-one uses it don’t need to worry about scaling don’t need to provision and manage servers

9. @theburningmonk theburningmonk.com in other words, it’s a lot like taking a cab

10. @theburningmonk theburningmonk.com Ownership Fuel Navigate To get there! Focus on getting there!

11. @theburningmonk theburningmonk.com HW Ownership OS Runtime & Scale Code Focus on getting there! Physical Servers Virtual Machines Containers Serverless

12. @theburningmonk theburningmonk.com Nano Services Self Managed Cost Paradigm ChangeAsync Dynamic agile env

13. “why are we failing at this?”

14. hidden dangers

15. @theburningmonk theburningmonk.com monolith microservices serverless

16. @theburningmonk theburningmonk.com monolith microservices serverless observability distributed systems bounded context

17. @theburningmonk theburningmonk.com monolith microservices serverless observability distributed systems bounded context

18. @theburningmonk theburningmonk.com monolith microservices serverless observability distributed systems bounded context event driven

19. @theburningmonk theburningmonk.com monolith serverless missing learnings from microservices

20. @theburningmonk theburningmonk.com monolith serverless missing learnings from microservices poor decisions

21. Yan Cui http://theburningmonk.com @theburningmonk AWS user for 10 years

22. http://bit.ly/yubl-serverless

23. Yan Cui http://theburningmonk.com @theburningmonk Developer Advocate @

24. Yan Cui http://theburningmonk.com @theburningmonk Independent Consultant advisetraining delivery

25. @theburningmonk theburningmonk.com https://theburningmonk.com/workshops Amsterdam, March 19-20 Helsinki, May 4-5 Stockholm, May 14-15 Dublin, June 16-17 London, September 24-25 Berlin, October 8-9

26. #1 not letting go of legacy thinking

27. “we’re doing serverless, but why aren’t thing going faster?”

28. @theburningmonk theburningmonk.com Socio Technical

29. @theburningmonk theburningmonk.com there are no silver bullets

31. @theburningmonk theburningmonk.com centralised team Team A Team B Team C Team D …

32. @theburningmonk theburningmonk.com “but the developers don’t understand AWS and how our infrastructure is set up”

33. @theburningmonk theburningmonk.com “but the developers don’t understand AWS and how our infrastructure is set up” let’s solve this problem instead!

34. @theburningmonk theburningmonk.com what got you here won’t get you there

35. @theburningmonk theburningmonk.com if (path == “/user” && method == “GET”) { return getUser(…); } else if (path == “/user” && method == “DELETE”) { return deleteUser(…); } else if (path == “/user” && method == “POST”) { return createUser(…); } else if …. Monolithic Functions

36. @theburningmonk theburningmonk.com GET /user POST /user DELETE /user Single-Purposed Functions

37. @theburningmonk theburningmonk.com author: yan.cui feature: user-api user-api-dev Monolithic Single-Purposed author: yan.cui feature: user-api user-api-dev-get-user author: yan.cui feature: user-api user-api-dev-create-user author: yan.cui feature: user-api user-api-dev-delete-user

38. @theburningmonk theburningmonk.com author: yan.cui feature: user-api user-api-dev Monolithic Single-Purposed author: yan.cui feature: user-api user-api-dev-get-user author: yan.cui feature: user-api user-api-dev-create-user author: yan.cui feature: user-api user-api-dev-delete-user ﬁnd related functions by preﬁx

39. @theburningmonk theburningmonk.com author: yan.cui feature: user-api user-api-dev Monolithic Single-Purposed author: yan.cui feature: user-api user-api-dev-get-user author: yan.cui feature: user-api user-api-dev-create-user author: yan.cui feature: user-api user-api-dev-delete-user discoverability (without having to dig into the code)

40. @theburningmonk theburningmonk.com author: yan.cui feature: user-api user-api-dev Monolithic Single-Purposed author: yan.cui feature: user-api user-api-dev-get-user author: yan.cui feature: user-api user-api-dev-create-user author: yan.cui feature: user-api user-api-dev-delete-user what does it do?

41. @theburningmonk theburningmonk.com author: yan.cui feature: user-api user-api-dev Monolithic Single-Purposed author: yan.cui feature: user-api user-api-dev-get-user author: yan.cui feature: user-api user-api-dev-create-user author: yan.cui feature: user-api user-api-dev-delete-user dynamodb:GetItem dynamodb:PutItem dynamodb:DeleteItem

42. @theburningmonk theburningmonk.com author: yan.cui feature: user-api user-api-dev Monolithic Single-Purposed author: yan.cui feature: user-api user-api-dev-get-user author: yan.cui feature: user-api user-api-dev-create-user author: yan.cui feature: user-api user-api-dev-delete-user dynamodb:GetItem dynamodb:PutItem dynamodb:DeleteItem no least privilege…

43. @theburningmonk theburningmonk.com author: yan.cui feature: user-api user-api-dev Monolithic Single-Purposed author: yan.cui feature: user-api user-api-dev-get-user author: yan.cui feature: user-api user-api-dev-create-user author: yan.cui feature: user-api user-api-dev-delete-user require(x) require(y) require(z)

44. @theburningmonk theburningmonk.com author: yan.cui feature: user-api user-api-dev Monolithic Single-Purposed author: yan.cui feature: user-api user-api-dev-get-user author: yan.cui feature: user-api user-api-dev-create-user author: yan.cui feature: user-api user-api-dev-delete-user require(x) require(y) require(z)

46. @theburningmonk theburningmonk.com more dependecies equals slower cold start

47. @theburningmonk theburningmonk.com author: yan.cui feature: user-api user-api-dev Monolithic Single-Purposed author: yan.cui feature: user-api user-api-dev-get-user author: yan.cui feature: user-api user-api-dev-create-user author: yan.cui feature: user-api user-api-dev-delete-user require(x) require(y) require(z) worse cold start performance

48. @theburningmonk theburningmonk.com keep functions simple, and single-purposed

49. #2 one account that rules them all

50. @theburningmonk theburningmonk.com mind the shared limits

51. @theburningmonk theburningmonk.com no. of DynamoDB tables no. of API Gateway regional APIs no. of API Gateway edge-optimized APIs no. of Kinesis shards no. of IAM roles no. of S3 buckets no. of CloudFormation stacks no. of SNS subscription ﬁlters no. of SSM parameters … Resource Limits

52. @theburningmonk theburningmonk.com DynamoDB read & write API Gateway requests/second Lambda concurrent executions SSM parameter ops/second … Throughput Limits

54. @theburningmonk theburningmonk.com compartmentalise security breaches

55. @theburningmonk theburningmonk.com One account per Team per Environment

57. #3 do first, research later

58. @theburningmonk theburningmonk.com https://einaregilsson.com/serverless-15-percent-slower-and-eight-times-more-expensive/

62. @theburningmonk theburningmonk.com the platforms need to do better at educating users on how to choose between different services

63. @theburningmonk theburningmonk.com SNS vs SQS vs Kinesis vs MKS? the platforms need to do better at educating users on how to choose between different services

64. @theburningmonk theburningmonk.com ordering replay events Kinesis SQS SNS by shard none (standard) global (FIFO) none up to 7 days none none mode retry batched batched (up to 10) singular retried until success (customizable) retry + DLQ retry + DLQ concurrency 1 per shard auto-scaled fan-out!!! subscribers many one-to-one many EventBridge many none none singular retry + DLQ fan-out!!!

65. @theburningmonk theburningmonk.com https://medium.com/theburningmonk-com/all-my-posts-on-serverless-aws-lambda-43c17a147f91

66. @theburningmonk theburningmonk.com https://www.jeremydaly.com/newsletter/

67. #4 not using a deployment toolkit

69. @theburningmonk theburningmonk.com https://lumigo.io/blog/comparison-of-lambda-deployment-frameworks/

70. @theburningmonk theburningmonk.com don’t write your own deployment framework

71. #5 console-driven development

74. #6 one repo per function

75. @theburningmonk theburningmonk.com github repo github repo github repo github repo github repo github repo github repo github repo github repo

76. @theburningmonk theburningmonk.com github repo github repo github repo github repo github repo github repo github repo github repo github repo

77. @theburningmonk theburningmonk.com monorepo?

78. @theburningmonk theburningmonk.com github repo

79. @theburningmonk theburningmonk.com one repo per service?

80. @theburningmonk theburningmonk.com github repo github repo github repo github repo user-api timeline-api relationship-api search-api

81. @theburningmonk theburningmonk.com https://lumigo.io/blog/mono-repo-vs-one-per-service/

89. @theburningmonk theburningmonk.com github repo github repo github repo github repo user-api timeline-api relationship-api search-api

90. @theburningmonk theburningmonk.com CI/CD pipeline per service

91. @theburningmonk theburningmonk.com functions are deployed together, as a stack

92. unencrypted secrets in env vars #7

93. @theburningmonk theburningmonk.com secrets should NEVER be in plain text in env variables

94. @theburningmonk theburningmonk.com SSM Parameter Store Secret 1 Secret 2 IAM Environment: SECRET_1: … SECRET_2: … Environment: SECRET_1: … SECRET_2: …

95. @theburningmonk theburningmonk.com SSM Parameter Store Secret 1 Secret 2 IAM Environment: SECRET_1: … SECRET_2: … Environment: SECRET_1: … SECRET_2: … yay!

98. @theburningmonk theburningmonk.com SSM Parameter Store Secret 1 Secret 2 IAM fetch at cold start, cache, invalidate every x mins

99. @theburningmonk theburningmonk.com https://github.com/middyjs/middy

101. @theburningmonk theburningmonk.com SSM Parameter Store Secret 1 Secret 2 IAM switch to Higher Throughput if you need more than 40 ops/s

102. not following least privilege principle #8

105. missing DLQs #9

106. @theburningmonk theburningmonk.com async sync S3 SNS SES CloudFormation CloudWatch Logs CloudWatch Events Scheduled Events CodeCommit AWS Conﬁg http://amzn.to/2v7Kc3b Cognito Alexa Lex API Gateway pulling DynamoDB Stream Kinesis Stream SQS

107. @theburningmonk theburningmonk.com async sync S3 SNS SES CloudFormation CloudWatch Logs CloudWatch Events Scheduled Events CodeCommit AWS Conﬁg http://amzn.to/2vs2lIg Cognito Alexa Lex API Gateway pulling DynamoDB Stream Kinesis Stream SQS Lambda handles retries (twice, then DLQ)

108. @theburningmonk theburningmonk.com conﬁgure DLQ for async functions so you don’t lose failed events

111. @theburningmonk theburningmonk.com DLQ Lambda Destinations payload payload, context(s), and response

113. too much/too little concurrency #10

114. @theburningmonk theburningmonk.com “Lambda generates too much load for the downstream system”

115. @theburningmonk theburningmonk.com one invocation per message SNS Lambda

116. @theburningmonk theburningmonk.com Downstream System SNS Lambda

118. @theburningmonk theburningmonk.com if you want… maximum throughput SNS precise control over throughput Kinesis

119. @theburningmonk theburningmonk.com if you want… maximum throughput SNS precise control over throughput Kinesis how quickly it scales out

120. @theburningmonk theburningmonk.com if you want… maximum throughput SNS precise control over throughput Kinesis how quickly it scales out SQS DynamoDB Streams

122. cold starts #11

123. @theburningmonk theburningmonk.com “cold starts only happen to the ﬁrst request”

124. @theburningmonk theburningmonk.com function invocationconcurrent execution i.e. a container

125. @theburningmonk theburningmonk.com function invocationconcurrent execution i.e. a container class instance method call

126. @theburningmonk theburningmonk.com Lambda scales the number of concurrent executions based on trafﬁc

127. @theburningmonk theburningmonk.com existing “containers” are reused where possible

128. @theburningmonk theburningmonk.com time invocation

129. @theburningmonk theburningmonk.com time invocation invocation

130. @theburningmonk theburningmonk.com time invocation invocation

131. @theburningmonk theburningmonk.com time invocation invocation invocation invocation

132. @theburningmonk theburningmonk.com time invocation invocation invocation invocation invocation invocation

133. @theburningmonk theburningmonk.com time invocation invocation invocation invocation invocation invocation

134. @theburningmonk theburningmonk.com time invocation invocation invocation invocation invocation invocation invocation

135. @theburningmonk theburningmonk.com time invocation invocation invocation invocation invocation invocation invocation invocation

142. @theburningmonk theburningmonk.com time invocation invocation ping invocation invocation invocation ping ping

143. @theburningmonk theburningmonk.com Lambda warmers don’t work when you have > 1 concurrent executions

144. @theburningmonk theburningmonk.com FREQUENCY DURATION

145. @theburningmonk theburningmonk.com FREQUENCY DURATION dictated by user trafﬁc, out of your control

146. @theburningmonk theburningmonk.com cold starts is generally not an issue if you have a steady trafﬁc pattern

147. @theburningmonk theburningmonk.com time req/s

148. @theburningmonk theburningmonk.com time req/s El Classico

149. @theburningmonk theburningmonk.com time req/s lunch dinner

150. @theburningmonk theburningmonk.com FREQUENCY DURATION optimize this!

151. @theburningmonk theburningmonk.com minimise the duration of cold starts so they fall within acceptable latency range

152. @theburningmonk theburningmonk.com time req/s lunch dinner Provisioned Concurrency

153. @theburningmonk theburningmonk.com time req/s lunch dinner Provisioned Concurrency On-Demand Concurrency

154. @theburningmonk theburningmonk.com https://lumigo.io/blog/provisioned-concurrency-the-end-of-cold-starts/

155. @theburningmonk theburningmonk.com there are no silver bullets

156. @theburningmonk theburningmonk.com reserved concurrency is a powerful tool IFF you have a cold start problem don’t use it by default

157. RDS connection handling #12

158. @theburningmonk theburningmonk.com default RDS conﬁgs are bad for Lambda

159. @theburningmonk theburningmonk.com default RDS conﬁgs are bad for Lambda idle connections are not closed too many connections per “container” max open connection is too low

160. @theburningmonk theburningmonk.com https://www.jeremydaly.com/manage-rds-connections-aws-lambda/

161. @theburningmonk theburningmonk.com set “wait_timeout” and “interactive_timeout” to 10 mins (default is 8 hours!)

162. @theburningmonk theburningmonk.com increase “max_connections” setting

163. @theburningmonk theburningmonk.com set client socket pool size to 1

166. (lack of) observability #13

167. @theburningmonk theburningmonk.com happened system repaireduser impact reduce MTTR

168. @theburningmonk theburningmonk.com Identify & Resolve Issues Understanding costs Visibility

169. @theburningmonk theburningmonk.com Identify & Resolve Issues Understanding costs Visibility

170. @theburningmonk theburningmonk.com happened system repaireduser impact MTTDiscovery

172. @theburningmonk theburningmonk.com “What alerts should I have?”

173. @theburningmonk theburningmonk.com It depends on what you’re building…

174. @theburningmonk theburningmonk.com But, this is a good starting point

175. @theburningmonk theburningmonk.com Lambda error rate % throttle count DLR error count iterator age regional concurrency

176. @theburningmonk theburningmonk.com Lambda error rate % throttle count DLR error count iterator age regional concurrency API Gateway p90/95/99 latency success rate % 4xx rate % 5xx rate %

177. @theburningmonk theburningmonk.com API Gateway p90/95/99 latency success rate % 4xx rate % 5xx rate % SQS message age Lambda error rate % throttle count DLR error count iterator age regional concurrency

178. @theburningmonk theburningmonk.com API Gateway p90/95/99 latency success rate % 4xx rate % 5xx rate % SQS message age Step Functions failed count throttle count timed out count Lambda error rate % throttle count DLR error count iterator age regional concurrency

179. @theburningmonk theburningmonk.com SQS message age Step Functions failed count throttle count timed out count API Gateway p90/95/99 latency success rate % 4xx rate % 5xx rate % Lambda error rate % throttle count DLR error count iterator age regional concurrency

180. @theburningmonk theburningmonk.com monitor and alert on message ﬂow rate for event processing pipelines

181. @theburningmonk theburningmonk.com “Can’t you codify these?”

183. https://theburningmonk.com/hire-me AdviseTraining Delivery “Fundamentally, Yan has improved our team by increasing our ability to derive value from AWS and Lambda in particular.” Nick Blair Tech Lead

184. @theburningmonk theburningmonk.com https://theburningmonk.com/workshops Amsterdam, March 19-20 Helsinki, May 4-5 Stockholm, May 14-15 Dublin, June 16-17 London, September 24-25 Berlin, October 8-9 codemotion-2020 10% off with code

185. Production-Ready Serverless

186. @theburningmonk theburningmonk.com github.com/theburningmonk