Lambda and DynamoDB best practices
•
2 likes•821 views
This document contains a summary of best practices for using Lambda and DynamoDB presented by Yan Cui. Some key recommendations include implementing observability from the start by using metrics and alerts to monitor performance, creating separate AWS accounts for each team and environment, loading secrets securely from SSM Parameter Store at runtime, following the principle of least privilege with IAM policies, parallelizing functions where possible, and various DynamoDB optimizations like using DocumentClient and PAY_PER_REQUEST billing. The document emphasizes that best practices depend on individual contexts and situations.
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Lambda and DynamoDB best practices
Similar to Lambda and DynamoDB best practices (20)
More from Yan Cui
More from Yan Cui (20)
Recently uploaded
Recently uploaded (20)
Lambda and DynamoDB best practices
- 1. Lambda & DynamoDB Best Practices a talk by Yan Cui
- 2. “Best practice is usually just someone else’s opinion” - random person on the internet
- 3. The “goodness” of a practice is tied to the context in which it is applied
- 5. “Good ideas that work for most people, most of the time”
- 6. Yan Cui http://theburningmonk.com @theburningmonk AWS user for 10 years
- 8. Yan Cui http://theburningmonk.com @theburningmonk Independent Consultant advise training delivery
- 9. Yan Cui http://theburningmonk.com @theburningmonk running serverless in production since 2016
- 10. 01. Observability from the start
- 11. A measure of how well the internal state of a system can be inferred from its external outputs Observability
- 12. happens…
- 13. everything fails, all the time
- 14. happened system repaired user impact reduce MTTR
- 15. Identify & Resolve Issues
- 16. Identify & Resolve Issues
- 17. happened system repaired user impact MTTDiscovery
- 19. “What alerts should I have?”
- 20. It depends on what you’re building…
- 21. Lambda error rate % throttle count
- 22. Lambda error rate % throttle count DLQ error count iterator age
- 23. Lambda error rate % throttle count DLQ error count iterator age regional concurrency
- 24. Lambda error rate % throttle count DLQ error count iterator age regional concurrency API Gateway p90/95/99 latency success rate % 4xx rate % 5xx rate %
- 25. API Gateway p90/95/99 latency success rate % 4xx rate % 5xx rate % SQS message age Lambda error rate % throttle count DLQ error count iterator age regional concurrency
- 26. happened system repaired user impact finding root cause
- 28. the needle is here somewhere…
- 29. This is my approach nowadays
- 30. + high-value structured logs + metrics + alerts
- 31. + high-value structured logs + metrics + alerts most of my troubleshooting
- 33. errors are captured and categorized
- 34. errors are captured and categorized frequency and trends
- 36. did errors correlate to a deployment?
- 41. invocation event, env vars, logs, etc.
- 47. + high-value structured logs + metrics + alerts Lambda invocations + every IO-request
- 48. + high-value structured logs + metrics + alerts Lambda invocations + every IO-request complex (non-IO) biz logic
- 50. logs from all the functions
- 51. + high-value structured logs + metrics + alerts system metrics for AWS services
- 53. 02. One account per team per environment
- 54. Mind the shared limits
- 55. no. of DynamoDB tables no. of API Gateway regional APIs no. of API Gateway edge-optimized APIs no. of Kinesis shards no. of IAM roles no. of S3 buckets no. of CloudFormation stacks no. of SNS subscription filters no. of SSM parameters … Resource Limits
- 56. DynamoDB read & write API Gateway requests/second Lambda concurrent executions SSM parameter ops/second … Throughput Limits
- 59. One account per Team per Environment
- 60. Isolate critical/high-throughput services to their own accounts
- 62. org-formation
- 64. org-formation
- 65. org-formation
- 66. org-formation
- 67. org-formation
- 68. org-formation
- 70. org-formation
- 76. 03. Load secrets at runtime ? ? ? ? ? ? ? ? ? ? ? ?? ? ? ?
- 77. SSM Parameter Store Secret 1 Secret 2
- 78. SSM Parameter Store Secret 1 Secret 2 IAM Environment: SECRET_1: … SECRET_2: … Environment: SECRET_1: … SECRET_2: …
- 79. SSM Parameter Store Secret 1 Secret 2 IAM Environment: SECRET_1: … SECRET_2: … Environment: SECRET_1: … SECRET_2: … yay!
- 82. Secrets should NEVER be in plain text in env variables
- 83. SSM Parameter Store IAM fetch at cold start, cache, invalidate every x mins Secret 1 Secret 2
- 86. SSM Parameter Store IAM Secret 1 Secret 2 switch to Higher Throughput for production
- 87. Secrets Manager IAM Secret 1 Secret 2 built-in rotation, more expensive
- 88. 04. Principle of least privilege
- 94. network boundary full-trust zero-trust networking
- 95. network boundary full-trust zero-trust networking compromised nodes give attackers access to our entire system
- 96. trust no-one
- 97. trust no-one authenticate and authorize every request
- 98. trust no-one authenticate and authorize every request use IAM to protect internal APIs
- 99. network security is a bonus, not the only line of defense
- 100. 05. Parallelise where you can
- 102. No dependency
- 104. faster!
- 105. faster! cheaper!
- 106. 06. Quick wins
- 107. Set environment variable AWS_NODEJS_CONNECTION_REUSE_ENABLED to “1” (for Node.js function running AWS SDK v1.x)
- 108. Use Database Proxies when working with RDS
- 109. Smaller deployment artefact === faster coldstart
- 110. Adding more memory DOESN’T help reduce cold start duration (except for JVM functions)
- 112. Use Lambda Layers as a deployment optimization
- 113. NOT as a package manager Use Lambda Layers as a deployment optimization
- 117. const AWS = require(‘aws-sdk’) (for Node.js function running AWS SDK v1.x) const DynamoDB = require(‘aws-sdk/clients/dynamodb’)
- 118. Prefer Lambda Destination over DLQs
- 120. DLQ Lambda Destinations payload payload, context(s), and response
- 122. 07. DynamoDB
- 123. Use DocumentClient instead of AWS.DynamoDB (for Node.js function running AWS SDK v1.x)
- 124. Use PAY_PER_REQUEST billing mode as default
- 125. Store large blobs in S3
- 126. Use BatchGetItem and BatchWriteItem to read/write multiple items
- 127. Avoid Scan unless you absolutely have to
- 128. Use caching to avoid DynamoDB calls
- 129. Use high cardinality keys as hash key
- 130. Use ULIDs as sort key
- 133. Use SSE with KMS CMK
- 135. Learn single-table design patterns
- 137. Learn single-table design patterns But don’t turn it into a religion
- 138. single-table design Steep learning curve.
- 139. single-table design Steep learning curve. Difficult to add new access patterns.
- 140. single-table design Steep learning curve. Difficult to add new access patterns. Can’t monitor usage cost by entity type.
- 141. single-table design Steep learning curve. Difficult to add new access patterns. Can’t monitor usage cost by entity type. Difficult to use DynamoDB streams.
- 142. “But what about all the cost savings from Single-Table Design?!”
- 143. “But what about all the cost savings from Single-Table Design?!” Only matters when running at scale.
- 144. The “goodness” of a practice is tied to the context in which it is applied
- 145. A best practice for Amazon is probably not best for you.
- 147. https://theburningmonk.com/hire-me Advise Training Delivery “Fundamentally, Yan has improved our team by increasing our ability to derive value from AWS and Lambda in particular.” Nick Blair Tech Lead