1. Lambda &
DynamoDB
Best Practices
a talk by Yan Cui

2. “Best practice is usually just someone else’s opinion”
- random person on the internet

3. The “goodness” of a practice is tied to
the context in which it is applied

5. “Good ideas that work for most people, most of the time”

6. Yan Cui
http://theburningmonk.com
@theburningmonk
AWS user for 10 years

7. Yan Cui
http://theburningmonk.com
@theburningmonk
Developer Advocate @

8. Yan Cui
http://theburningmonk.com
@theburningmonk
Independent Consultant
advise
training delivery

9. Yan Cui
http://theburningmonk.com
@theburningmonk running serverless in
production since 2016

10. 01. Observability from the start

11. A measure of how well the internal state of a
system can be inferred from its external outputs
Observability

12. happens…

13. everything fails, all the time

14. happened system repaired
user impact
reduce MTTR

15. Identify & Resolve Issues

16. Identify & Resolve Issues

17. happened system repaired
user impact
MTTDiscovery

19. “What alerts should I have?”

20. It depends on what you’re building…

21. Lambda
error rate %
throttle count

22. Lambda
error rate %
throttle count
DLQ error count
iterator age

23. Lambda
error rate %
throttle count
DLQ error count
iterator age
regional concurrency

24. Lambda
error rate %
throttle count
DLQ error count
iterator age
regional concurrency
API Gateway
p90/95/99 latency
success rate %
4xx rate %
5xx rate %

25. API Gateway
p90/95/99 latency
success rate %
4xx rate %
5xx rate %
SQS
message age
Lambda
error rate %
throttle count
DLQ error count
iterator age
regional concurrency

26. happened system repaired
user impact
finding root cause

27. Logs are over-rated

28. the needle is here
somewhere…

29. This is my approach nowadays

30. + high-value
structured logs +
metrics + alerts

31. + high-value
structured logs +
metrics + alerts
most of my
troubleshooting

33. errors are captured
and categorized

34. errors are captured
and categorized
frequency and trends

36. did errors correlate
to a deployment?

41. invocation event,
env vars, logs, etc.

47. + high-value
structured logs +
metrics + alerts
Lambda invocations +
every IO-request

48. + high-value
structured logs +
metrics + alerts
Lambda invocations +
every IO-request
complex (non-IO)
biz logic

49. logs and traces
side-by-side

50. logs from all the
functions

51. + high-value
structured logs +
metrics + alerts
system metrics for
AWS services

53. 02. One account per team per environment

54. Mind the shared limits

55. no. of DynamoDB tables
no. of API Gateway regional APIs
no. of API Gateway edge-optimized APIs
no. of Kinesis shards
no. of IAM roles
no. of S3 buckets
no. of CloudFormation stacks
no. of SNS subscription filters
no. of SSM parameters
…
Resource Limits

56. DynamoDB read & write
API Gateway requests/second
Lambda concurrent executions
SSM parameter ops/second
…
Throughput Limits

58. Compartmentalise security breaches

59. One account per Team per Environment

60. Isolate critical/high-throughput services
to their own accounts

62. org-formation

63. org-formation
infrastructure-as-code
CloudFormation-like YML syntax
template landing zones

64. org-formation

65. org-formation

66. org-formation

67. org-formation

68. org-formation

69. org-formation
> org-formation update

70. org-formation

74. org-formation
> org-formation perform-tasks

75. org-formation
https://github.com/OlafConijn/AwsOrganizationFormation

76. 03. Load secrets
at runtime
?
?
?
?
?
?
?
?
?
? ?
??
?
?
?

77. SSM Parameter Store
Secret 1
Secret 2

78. SSM Parameter Store
Secret 1
Secret 2
IAM
Environment:
SECRET_1: …
SECRET_2: …
Environment:
SECRET_1: …
SECRET_2: …

79. SSM Parameter Store
Secret 1
Secret 2
IAM
Environment:
SECRET_1: …
SECRET_2: …
Environment:
SECRET_1: …
SECRET_2: …
yay!

82. Secrets should NEVER be in plain text in env variables

83. SSM Parameter Store IAM
fetch at cold start,
cache,
invalidate every x mins
Secret 1
Secret 2

84. https://github.com/middyjs/middy

86. SSM Parameter Store IAM
Secret 1
Secret 2
switch to Higher
Throughput for production

87. Secrets Manager IAM
Secret 1
Secret 2
built-in rotation,
more expensive

88. 04. Principle of
least privilege

91. Zero-trust networking

92. network boundary
full-trust

93. network boundary
full-trust

94. network boundary
full-trust zero-trust networking

95. network boundary
full-trust zero-trust networking
compromised nodes give attackers
access to our entire system

96. trust no-one

97. trust no-one
authenticate and
authorize every request

98. trust no-one
authenticate and
authorize every request
use IAM to protect
internal APIs

99. network security is a bonus, not the only line of defense

100. 05. Parallelise
where you can

102. No dependency

104. faster!

105. faster!
cheaper!

106. 06. Quick wins

107. Set environment variable
AWS_NODEJS_CONNECTION_REUSE_ENABLED
to “1”
(for Node.js function running AWS SDK v1.x)

108. Use Database Proxies when working with RDS

109. Smaller deployment artefact === faster coldstart

110. Adding more memory DOESN’T help reduce cold start duration
(except for JVM functions)

111. Trim your depedencies

112. Use Lambda Layers as a deployment optimization

113. NOT as a package manager
Use Lambda Layers as a deployment optimization

117. const AWS = require(‘aws-sdk’)
(for Node.js function running AWS SDK v1.x)
const DynamoDB = require(‘aws-sdk/clients/dynamodb’)

118. Prefer Lambda Destination over DLQs

119. DLQ Lambda Destinations
payload

120. DLQ Lambda Destinations
payload payload, context(s), and response

122. 07. DynamoDB

123. Use DocumentClient instead of AWS.DynamoDB
(for Node.js function running AWS SDK v1.x)

124. Use PAY_PER_REQUEST billing mode as default

125. Store large blobs in S3

126. Use BatchGetItem and BatchWriteItem to
read/write multiple items

127. Avoid Scan unless you absolutely have to

128. Use caching to avoid DynamoDB calls

129. Use high cardinality keys as hash key

130. Use ULIDs as sort key

133. Use SSE with KMS CMK

134. Enable point-in-time recovery

135. Learn single-table design patterns

136. gumroad.com/a/279377011

137. Learn single-table design patterns
But don’t turn it into a religion

138. single-table design
Steep learning curve.

139. single-table design
Steep learning curve.
Difficult to add new access patterns.

140. single-table design
Steep learning curve.
Difficult to add new access patterns.
Can’t monitor usage cost by entity type.

141. single-table design
Steep learning curve.
Difficult to add new access patterns.
Can’t monitor usage cost by entity type.
Difficult to use DynamoDB streams.

142. “But what about all the cost savings from Single-Table Design?!”

143. “But what about all the cost savings from Single-Table Design?!”
Only matters when running at scale.

144. The “goodness” of a practice is tied to
the context in which it is applied

145. A best practice for Amazon is probably not best for you.

147. https://theburningmonk.com/hire-me
Advise
Training Delivery
“Fundamentally, Yan has improved our team by increasing our
ability to derive value from AWS and Lambda in particular.”
Nick Blair
Tech Lead

148. @theburningmonk
theburningmonk.com
github.com/theburningmonk