Successfully reported this slideshow.
Your SlideShare is downloading. ×

Lambda and DynamoDB best practices

Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad

Check these out next

1 of 148 Ad
Advertisement

More Related Content

Slideshows for you (20)

Similar to Lambda and DynamoDB best practices (20)

Advertisement

More from Yan Cui (20)

Recently uploaded (20)

Advertisement

Lambda and DynamoDB best practices

  1. 1. Lambda & DynamoDB Best Practices a talk by Yan Cui
  2. 2. “Best practice is usually just someone else’s opinion” - random person on the internet
  3. 3. The “goodness” of a practice is tied to the context in which it is applied
  4. 4. “Good ideas that work for most people, most of the time”
  5. 5. Yan Cui http://theburningmonk.com @theburningmonk AWS user for 10 years
  6. 6. Yan Cui http://theburningmonk.com @theburningmonk Developer Advocate @
  7. 7. Yan Cui http://theburningmonk.com @theburningmonk Independent Consultant advise training delivery
  8. 8. Yan Cui http://theburningmonk.com @theburningmonk running serverless in production since 2016
  9. 9. 01. Observability from the start
  10. 10. A measure of how well the internal state of a system can be inferred from its external outputs Observability
  11. 11. happens…
  12. 12. everything fails, all the time
  13. 13. happened system repaired user impact reduce MTTR
  14. 14. Identify & Resolve Issues
  15. 15. Identify & Resolve Issues
  16. 16. happened system repaired user impact MTTDiscovery
  17. 17. “What alerts should I have?”
  18. 18. It depends on what you’re building…
  19. 19. Lambda error rate % throttle count
  20. 20. Lambda error rate % throttle count DLQ error count iterator age
  21. 21. Lambda error rate % throttle count DLQ error count iterator age regional concurrency
  22. 22. Lambda error rate % throttle count DLQ error count iterator age regional concurrency API Gateway p90/95/99 latency success rate % 4xx rate % 5xx rate %
  23. 23. API Gateway p90/95/99 latency success rate % 4xx rate % 5xx rate % SQS message age Lambda error rate % throttle count DLQ error count iterator age regional concurrency
  24. 24. happened system repaired user impact finding root cause
  25. 25. Logs are over-rated
  26. 26. the needle is here somewhere…
  27. 27. This is my approach nowadays
  28. 28. + high-value structured logs + metrics + alerts
  29. 29. + high-value structured logs + metrics + alerts most of my troubleshooting
  30. 30. errors are captured and categorized
  31. 31. errors are captured and categorized frequency and trends
  32. 32. did errors correlate to a deployment?
  33. 33. invocation event, env vars, logs, etc.
  34. 34. + high-value structured logs + metrics + alerts Lambda invocations + every IO-request
  35. 35. + high-value structured logs + metrics + alerts Lambda invocations + every IO-request complex (non-IO) biz logic
  36. 36. logs and traces side-by-side
  37. 37. logs from all the functions
  38. 38. + high-value structured logs + metrics + alerts system metrics for AWS services
  39. 39. 02. One account per team per environment
  40. 40. Mind the shared limits
  41. 41. no. of DynamoDB tables no. of API Gateway regional APIs no. of API Gateway edge-optimized APIs no. of Kinesis shards no. of IAM roles no. of S3 buckets no. of CloudFormation stacks no. of SNS subscription filters no. of SSM parameters … Resource Limits
  42. 42. DynamoDB read & write API Gateway requests/second Lambda concurrent executions SSM parameter ops/second … Throughput Limits
  43. 43. Compartmentalise security breaches
  44. 44. One account per Team per Environment
  45. 45. Isolate critical/high-throughput services to their own accounts
  46. 46. org-formation
  47. 47. org-formation infrastructure-as-code CloudFormation-like YML syntax template landing zones
  48. 48. org-formation
  49. 49. org-formation
  50. 50. org-formation
  51. 51. org-formation
  52. 52. org-formation
  53. 53. org-formation > org-formation update
  54. 54. org-formation
  55. 55. org-formation > org-formation perform-tasks
  56. 56. org-formation https://github.com/OlafConijn/AwsOrganizationFormation
  57. 57. 03. Load secrets at runtime ? ? ? ? ? ? ? ? ? ? ? ?? ? ? ?
  58. 58. SSM Parameter Store Secret 1 Secret 2
  59. 59. SSM Parameter Store Secret 1 Secret 2 IAM Environment: SECRET_1: … SECRET_2: … Environment: SECRET_1: … SECRET_2: …
  60. 60. SSM Parameter Store Secret 1 Secret 2 IAM Environment: SECRET_1: … SECRET_2: … Environment: SECRET_1: … SECRET_2: … yay!
  61. 61. Secrets should NEVER be in plain text in env variables
  62. 62. SSM Parameter Store IAM fetch at cold start, cache, invalidate every x mins Secret 1 Secret 2
  63. 63. https://github.com/middyjs/middy
  64. 64. SSM Parameter Store IAM Secret 1 Secret 2 switch to Higher Throughput for production
  65. 65. Secrets Manager IAM Secret 1 Secret 2 built-in rotation, more expensive
  66. 66. 04. Principle of least privilege
  67. 67. Zero-trust networking
  68. 68. network boundary full-trust
  69. 69. network boundary full-trust
  70. 70. network boundary full-trust zero-trust networking
  71. 71. network boundary full-trust zero-trust networking compromised nodes give attackers access to our entire system
  72. 72. trust no-one
  73. 73. trust no-one authenticate and authorize every request
  74. 74. trust no-one authenticate and authorize every request use IAM to protect internal APIs
  75. 75. network security is a bonus, not the only line of defense
  76. 76. 05. Parallelise where you can
  77. 77. No dependency
  78. 78. faster!
  79. 79. faster! cheaper!
  80. 80. 06. Quick wins
  81. 81. Set environment variable AWS_NODEJS_CONNECTION_REUSE_ENABLED to “1” (for Node.js function running AWS SDK v1.x)
  82. 82. Use Database Proxies when working with RDS
  83. 83. Smaller deployment artefact === faster coldstart
  84. 84. Adding more memory DOESN’T help reduce cold start duration (except for JVM functions)
  85. 85. Trim your depedencies
  86. 86. Use Lambda Layers as a deployment optimization
  87. 87. NOT as a package manager Use Lambda Layers as a deployment optimization
  88. 88. const AWS = require(‘aws-sdk’) (for Node.js function running AWS SDK v1.x) const DynamoDB = require(‘aws-sdk/clients/dynamodb’)
  89. 89. Prefer Lambda Destination over DLQs
  90. 90. DLQ Lambda Destinations payload
  91. 91. DLQ Lambda Destinations payload payload, context(s), and response
  92. 92. 07. DynamoDB
  93. 93. Use DocumentClient instead of AWS.DynamoDB (for Node.js function running AWS SDK v1.x)
  94. 94. Use PAY_PER_REQUEST billing mode as default
  95. 95. Store large blobs in S3
  96. 96. Use BatchGetItem and BatchWriteItem to read/write multiple items
  97. 97. Avoid Scan unless you absolutely have to
  98. 98. Use caching to avoid DynamoDB calls
  99. 99. Use high cardinality keys as hash key
  100. 100. Use ULIDs as sort key
  101. 101. Use SSE with KMS CMK
  102. 102. Enable point-in-time recovery
  103. 103. Learn single-table design patterns
  104. 104. gumroad.com/a/279377011
  105. 105. Learn single-table design patterns But don’t turn it into a religion
  106. 106. single-table design Steep learning curve.
  107. 107. single-table design Steep learning curve. Difficult to add new access patterns.
  108. 108. single-table design Steep learning curve. Difficult to add new access patterns. Can’t monitor usage cost by entity type.
  109. 109. single-table design Steep learning curve. Difficult to add new access patterns. Can’t monitor usage cost by entity type. Difficult to use DynamoDB streams.
  110. 110. “But what about all the cost savings from Single-Table Design?!”
  111. 111. “But what about all the cost savings from Single-Table Design?!” Only matters when running at scale.
  112. 112. The “goodness” of a practice is tied to the context in which it is applied
  113. 113. A best practice for Amazon is probably not best for you.
  114. 114. https://theburningmonk.com/hire-me Advise Training Delivery “Fundamentally, Yan has improved our team by increasing our ability to derive value from AWS and Lambda in particular.” Nick Blair Tech Lead
  115. 115. @theburningmonk theburningmonk.com github.com/theburningmonk

×