This document discusses best practices for preparing serverless applications on AWS Lambda for production. It describes the author's experience deploying 170 Lambda functions and 1.2GB of code into production with a 95% cost savings compared to EC2. Key practices covered include testing, monitoring, logging, distributed tracing, CI/CD pipelines, configuration management, and security. The author advocates for building on principles over tools and shares several tools for serverless development.

1. in production
an experience report
an experience report
what you should know before you go to production
ServerlessServerless

2. http://theburningmonk.com
@theburningmonk

3. http://theburningmonk.com
@theburningmonk

8. apr, 2016

11. 10PM
traffic

12. 10PM
traffic
70-100x

13. low utilisation to leave room for spikes
EC2 scaling is slow, so scale earlier

14. lots of $$$ for unused resources

15. up to 30 mins for deployment
deployment required downtime

16. - Dan North
“lead time to someone saying
thank you is the only reputation
metric that matters.”

18. “what would good
look like for us?”

19. be small
be fast
have zero downtime
have no lock-step
DEPLOYMENTS SHOULD...

20. FEATURES SHOULD...
be deployable independently
be loosely-coupled

21. WE WANT TO...
minimise cost for unused resources
minimise ops effort
reduce tech mess
deliver visible improvements faster

22. nov, 2016

23. 170 Lambda functions in prod
1.2 GB deployment packages in prod
95% cost saving vs EC2
15x no. of prod releases per month

24. time
is a good fit

25. 1st function in prod!
time
is a good fit

26. ?
time
is a good fit
1st function in prod!

27. ALERTING
CI / CD
TESTING
LOGGING
MONITORING

28. Practices ToolsPrinciples
what is good? how to make it good? with what?

29. Principles outlast Tools

30. 170 functions
? ?
time
is a good fit
1st function in prod!

31. SECURITY
DISTRIBUTED TRACING
CONFIG MANAGEMENT

32. evolving the PLATFORM

33. rebuilt search

34. Legacy Monolith Amazon Kinesis Amazon Lambda
Amazon CloudSearch

35. Legacy Monolith Amazon Kinesis Amazon Lambda
Amazon CloudSearchAmazon API Gateway Amazon Lambda

36. new analytics pipeline

37. Legacy Monolith Amazon Kinesis Amazon Lambda
Google BigQuery

38. Legacy Monolith Amazon Kinesis Amazon Lambda
Google BigQuery
1 developer, 2 days
design production
(his 1st serverless project)

39. Legacy Monolith Amazon Kinesis Amazon Lambda
Google BigQuery
“nothing ever got done
this fast at Skype!”
- Chris Twamley

40. - Dan North
“lead time to someone saying
thank you is the only reputation
metric that matters.”

41. Rebuilt
with Lambda

49. Rebuilt
with Lambda

50. BigQuery

51. BigQuery

52. grapheneDB
BigQuery

53. grapheneDB
BigQuery

54. grapheneDB
BigQuery

55. getting PRODUCTION READY

57. http://serverless.com

58. https://github.com/awslabs/serverless-application-model

59. http://apex.run

60. https://apex.github.io/up

61. https://github.com/claudiajs/claudia

62. https://github.com/Miserlou/Zappa

63. http://gosparta.io

64. TESTING

66. 1.Unit
do our objects do the right thing?
are they easy to work with?
Level of Testing

68. 1.Unit
2.Integration
does our code work against code we
can’t change?
Level of Testing

69. handler

70. handler
test by invoking
the handler

71. Level of Testing
1.Unit
2.Integration
3.Acceptance
does the whole system work?

72. unit
integration
acceptance
feedback
confidence
Level of Testing

73. “…We find that tests that mock external
libraries often need to be complex to get
the code into the right state for the
functionality we need to exercise.
The mess in such tests is telling us that
the design isn’t right but, instead of fixing
the problem by improving the code, we
have to carry the extra complexity in
both code and test…”
Don’t Mock Types You Can’t Change

74. “…The second risk is that we have to be
sure that the behaviour we stub or mock
matches what the external library will
actually do…
Even if we get it right once, we have to
make sure that the tests remain valid
when we upgrade the libraries…”
Don’t Mock Types You Can’t Change

75. Don’t Mock Types You Can’t Change
Services

76. Paul Johnston
http://bit.ly/2t5viwK

78. Unit Tests

79. Unit Tests
Mock/Stub

80. what unit tests will not tell you…
is Lambda proxy
configured correctly?
is the API resources
configured correctly?
is IAM policy set
up correctly?
is the request mapping
set up correctly?
is the table created?
are we assuming the
correct schema?
is our request correct?

82. risk
integrate
external services
observation

85. Paul Johnston

86. “…Wherever possible, an acceptance
test should exercise the system
end-to-end without directly calling its
internal code.
An end-to-end test interacts with the
system only from the outside: through
its interface…”
Testing End-to-End

87. Legacy Monolith Amazon Kinesis Amazon Lambda
Amazon CloudSearchAmazon API Gateway Amazon Lambda

88. Legacy Monolith Amazon Kinesis Amazon Lambda
Amazon CloudSearchAmazon API Gateway Amazon Lambda
Test Input

89. Legacy Monolith Amazon Kinesis Amazon Lambda
Amazon CloudSearchAmazon API Gateway Amazon Lambda
Validate
Test Input

90. Integration
my code

91. End-to-End
my code

92. HOW
observation

95. CI + CD PIPELINE

96. Yan

97. “…We prefer to have the end-to-end tests
exercise both the system and the
process by which it’s built and
deployed…
This sounds like a lot of effort (it is), but
has to be done anyway repeatedly during
the software’s lifetime…”
Testing End-to-End

98. Yan

99. Jenkins build config deploys and
tests
unit + integration tests
deploy
acceptance tests

104. http://alistair.cockburn.us/Hexagonal+architecture

105. build.sh allows repeatable builds on both local & CI

107. Auto Auto Manual

109. LOGGING

111. 2016-07-12T12:24:37.571Z 994f18f9-482b-11e6-8668-53e4eab441ae
GOT is off air, what do I do now?

112. 2016-07-12T12:24:37.571Z 994f18f9-482b-11e6-8668-53e4eab441ae
GOT is off air, what do I do now?
UTC Timestamp API Gateway Request Id
your log message

113. Yan

114. CloudWatch Logs

115. CloudWatch Logs AWS Lambda ELK stack

116. …

117. CloudWatch Events

120. DISTRIBUTED TRACING

122. a user

123. where could the
problem be?

124. correlation IDs*
* eg. request-id, user-id, yubl-id, etc.

126. kinesis client
http client
sns client

131. MONITORING + ALERTING

133. • invocation Count
• error Count
• latency
• throttling
• granular to the minute
• support custom metrics

134. • same metrics as CW
• better dashboard
• support custom
metrics
https://www.datadoghq.com/blog/monitoring-lambda-functions-datadog/

138. my code

139. my code

140. internet internet
press button something happens
my code

142. 100ms
http://bit.ly/2EXPfbA

144. console.log(“hydrating yubls from db…”);
console.log(“fetching user info from user-api”);
console.log(“MONITORING|1489795335|27.4|latency|user-api-latency”);
console.log(“MONITORING|1489795335|8|count|yubls-served”);
metric
s
logs
timestamp metric value metric name
metric type

145. CloudWatch Logs AWS Lambda
ELK stack
logs
metrics
CloudWatch

147. CONFIG MANAGEMENT

150. Environment variables make it
hard to share configurations
across functions.
me

151. me
Environment variables make it
hard to implement fine-grained
access to sensitive info.

152. config service
goes here

156. SSM
Parameter
Store

159. SSM Parameter Store
HTTPS
role-based access
encrypted in-flight

160. SSM Parameter Store
encrypt
role-based access

161. SSM Parameter Store
encrypted at-rest

162. HTTPS
SSM Parameter Store
encrypted in-flight
role-based access

166. That’s it, folks!
Thank you all :-D

167. API Gateway and Kinesis
Authentication & authorisation (IAM, Cognito)
Testing
Running & Debugging functions locally
Log aggregation
Monitoring & Alerting
X-Ray
Correlation IDs
CI/CD
Performance and Cost optimisation
Error Handling
Configuration management
VPC
Security
Leading practices (API Gateway, Kinesis, Lambda)
Canary deployments
http://bit.ly/production-ready-serverless

168. @theburningmonk
theburningmonk.com
github.com/theburningmonk