Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

DevOps with Serverless

455 views

Published on

How does serverless change the practices of DevOps?

Published in: Technology
  • Be the first to comment

DevOps with Serverless

  1. 1. with
  2. 2. Yan Cui http://theburningmonk.com @theburningmonk Principal Engineer @
  3. 3. “Netflix for sports” offices in London, Leeds, Katowice and Tokyo
  4. 4. available on 30+ platforms
  5. 5. ~500,000 concurrent viewers
  6. 6. available in Austria, Switzerland, Germany, Japan, Canada and Italy USA coming soon ;-)
  7. 7. “Netflix for sports” offices in London, Leeds, Katowice and Tokyo We’re hiring! Visit engineering.dazn.com to learn more. follow @dazneng for updates about the engineering team.
  8. 8. AWS user for 10 years
  9. 9. 2009…
  10. 10. DevOps!
  11. 11. as in.. devs doing ops?
  12. 12. let’s rebrand the ops team
  13. 13. Time
  14. 14. Time it’s working!!!
  15. 15. a set of practices to reduce the time between committing a change to a system and the change being placed into normal production, while ensuring high quality
  16. 16. a set of practices to reduce the time between committing a change to a system and the change being placed into normal production, while ensuring high quality
  17. 17. DevOps toolchain
  18. 18. http://bit.ly/2MNoRaY
  19. 19. http://bit.ly/2MNoRaY
  20. 20. FOCUS
  21. 21. one of the biggest distraction for tech teams has been the endless tinkering of tools
  22. 22. simple. does the job. gets out of my way.
  23. 23. tools not gadgets
  24. 24. http://serverless.com/framework +
  25. 25. Code Test Deploy {}
  26. 26. Code Test Deploy {}
  27. 27. github repo github repo github repo github repo user-api timeline-api relationship-api search-api
  28. 28. Code Test Deploy {} infrastructure-as-code
  29. 29. Code Test Deploy {} infrastructure-as-code CloudFormation
  30. 30. Code Test Deploy {}
  31. 31. https://amzn.to/29Lxuzu
  32. 32. Level of Testing 1.Unit do our objects do the right thing? are they easy to work with? https://amzn.to/29Lxuzu
  33. 33. Level of Testing 1.Unit 2.Integration does our code work against code we can’t change? https://amzn.to/29Lxuzu
  34. 34. handler
  35. 35. handler test by invoking the handler
  36. 36. Level of Testing 1.Unit 2.Integration 3.Acceptance does the whole system work? https://amzn.to/29Lxuzu
  37. 37. Level of Testing unit integration acceptance feedback confidence
  38. 38. “…We find that tests that mock external libraries often need to be complex to get the code into the right state for the functionality we need to exercise. The mess in such tests is telling us that the design isn’t right but, instead of fixing the problem by improving the code, we have to carry the extra complexity in both code and test…” Don’t Mock Types You Can’t Change https://amzn.to/29Lxuzu
  39. 39. “…The second risk is that we have to be sure that the behaviour we stub or mock matches what the external library will actually do… Even if we get it right once, we have to make sure that the tests remain valid when we upgrade the libraries…” Don’t Mock Types You Can’t Change https://amzn.to/29Lxuzu
  40. 40. Don’t Mock Types You Can’t Change Services https://amzn.to/29Lxuzu
  41. 41. Paul Johnston The serverless approach to testing is different and may actually be easier. http://bit.ly/2t5viwK
  42. 42. LambdaAPI Gateway DynamoDB
  43. 43. LambdaAPI Gateway DynamoDB Unit Tests
  44. 44. LambdaAPI Gateway DynamoDB Unit Tests Mock/Stub
  45. 45. is our request correct? is the request mapping set up correctly?is the API resources configured correctly? are we assuming the correct schema? LambdaAPI Gateway DynamoDB is Lambda proxy configured correctly? is IAM policy set up correctly? is the table created? what unit tests will not tell you…
  46. 46. most Lambda functions are simple have single purpose, the risk of shipping broken software has largely shifted to how they integrate with external services observation
  47. 47. optimize towards shipping working software, even if it means slowing down your feedback loop…
  48. 48. …if a service can’t provide you with a relatively easy way to test the interface in reality, then you should consider using another one. Paul Johnston
  49. 49. “…Wherever possible, an acceptance test should exercise the system end-to- end without directly calling its internal code. An end-to-end test interacts with the system only from the outside: through its interface…” Testing End-to-End
  50. 50. Legacy Monolith Amazon Kinesis Amazon Lambda Amazon CloudSearchAmazon API Gateway Amazon Lambda
  51. 51. Legacy Monolith Amazon Kinesis Amazon Lambda Amazon CloudSearchAmazon API Gateway Amazon Lambda Test Input
  52. 52. Legacy Monolith Amazon Kinesis Amazon Lambda Amazon CloudSearchAmazon API Gateway Amazon Lambda Test Input Validate
  53. 53. integration tests exercise system’s Integration with its external dependencies my code
  54. 54. acceptance tests exercise system End-to-End from the outside my code
  55. 55. integration tests differ from acceptance tests only in HOW the Lambda functions are invoked observation
  56. 56. Code Test Deploy {}
  57. 57. Code Test Deploy {}
  58. 58. serverless.yml {} Code
  59. 59. {} Code serverless.yml
  60. 60. serverless.yml {} Code S3
  61. 61. {} Code serverless.yml S3 CloudFormation
  62. 62. {} Code serverless.yml S3 CloudFormation
  63. 63. request blue-green deployment
  64. 64. request blue-green deployment
  65. 65. request blue-green deployment
  66. 66. request blue-green deployment req/s auto-scaling
  67. 67. request blue-green deployment req/s auto-scaling us-east-1a us-east-1b us-east-1c multi-AZ
  68. 68. request blue-green deployment req/s auto-scaling us-east-1a us-east-1b us-east-1c multi-AZ
  69. 69. Code Test Deploy {}
  70. 70. simple. does the job. gets out of my way.
  71. 71. https://amzn.to/2wb9kbt multi-region active-active
  72. 72. canary deployment http://bit.ly/2OXd6fn
  73. 73. Code Test Deploy {}
  74. 74. CI/CD Monitor
  75. 75. CI/CD Monitor drone circleci
  76. 76. if [ "$1" = "deploy" ] && [ $# -eq 4 ]; then STAGE=$2 REGION=$3 PROFILE=$4 npm install AWS_PROFILE=$PROFILE 'node_modules/.bin/sls' deploy -s $STAGE -r $REGION elif [ "$1" = "int-test" ] && [ $# -eq 4 ]; then STAGE=$2 REGION=$3 PROFILE=$4 npm install AWS_PROFILE=$PROFILE npm run int-$STAGE elif [ "$1" = "acceptance-test" ] && [ $# -eq 4 ]; then STAGE=$2 REGION=$3 PROFILE=$4 npm install AWS_PROFILE=$PROFILE npm run acceptance-$STAGE else usage exit 1 fi
  77. 77. if [ "$1" = "deploy" ] && [ $# -eq 4 ]; then STAGE=$2 REGION=$3 PROFILE=$4 npm install AWS_PROFILE=$PROFILE 'node_modules/.bin/sls' deploy -s $STAGE -r $REGION elif [ "$1" = "int-test" ] && [ $# -eq 4 ]; then STAGE=$2 REGION=$3 PROFILE=$4 npm install AWS_PROFILE=$PROFILE npm run int-$STAGE elif [ "$1" = "acceptance-test" ] && [ $# -eq 4 ]; then STAGE=$2 REGION=$3 PROFILE=$4 npm install AWS_PROFILE=$PROFILE npm run acceptance-$STAGE else usage exit 1 fi install Serverless framework as dev dependency
  78. 78. if [ "$1" = "deploy" ] && [ $# -eq 4 ]; then STAGE=$2 REGION=$3 PROFILE=$4 npm install AWS_PROFILE=$PROFILE 'node_modules/.bin/sls' deploy -s $STAGE -r $REGION elif [ "$1" = "int-test" ] && [ $# -eq 4 ]; then STAGE=$2 REGION=$3 PROFILE=$4 npm install AWS_PROFILE=$PROFILE npm run int-$STAGE elif [ "$1" = "acceptance-test" ] && [ $# -eq 4 ]; then STAGE=$2 REGION=$3 PROFILE=$4 npm install AWS_PROFILE=$PROFILE npm run acceptance-$STAGE else usage exit 1 fi install Serverless framework as dev dependency mitigate version conflicts
  79. 79. if [ "$1" = "deploy" ] && [ $# -eq 4 ]; then STAGE=$2 REGION=$3 PROFILE=$4 npm install AWS_PROFILE=$PROFILE 'node_modules/.bin/sls' deploy -s $STAGE -r $REGION elif [ "$1" = "int-test" ] && [ $# -eq 4 ]; then STAGE=$2 REGION=$3 PROFILE=$4 npm install AWS_PROFILE=$PROFILE npm run int-$STAGE elif [ "$1" = "acceptance-test" ] && [ $# -eq 4 ]; then STAGE=$2 REGION=$3 PROFILE=$4 npm install AWS_PROFILE=$PROFILE npm run acceptance-$STAGE else usage exit 1 fi build.sh allows repeatable builds on both local & CI
  80. 80. Auto Auto Manual
  81. 81. CI/CD Monitor
  82. 82. CI/CD Monitor logging
  83. 83. 2016-07-12T12:24:37.571Z 994f18f9-482b-11e6-8668-53e4eab441ae GOT is off air, what do I do now?
  84. 84. 2016-07-12T12:24:37.571Z 994f18f9-482b-11e6-8668-53e4eab441ae GOT is off air, what do I do now? UTC Timestamp API Gateway Request Id your log message
  85. 85. Me Logs are not easily searchable in CloudWatch Logs.
  86. 86. CloudWatch Logs
  87. 87. AWS Lambda invokes AWS Lambda stdout asynchronously any log aggregation service CloudWatch Logs
  88. 88. CloudWatch Logs AWS Lambda AWS Lambda stdout any log aggregation service asynchronously invokes
  89. 89.
  90. 90. CloudWatch Events
  91. 91. CI/CD Monitor logging tracing
  92. 92. a user my followers didn’t receive my new post!
  93. 93. where could the problem be?
  94. 94. correlation IDs* * eg. request-id, user-id, yubl-id, etc.
  95. 95. wrap HTTP client & AWS SDK clients to forward captured correlation IDs
  96. 96. kinesis client http client sns client
  97. 97. use X-Ray for performance tracing
  98. 98. Amazon X-Ray
  99. 99. Amazon X-Ray
  100. 100. CI/CD Monitor logging tracing metrics
  101. 101. no place to install agents/daemons
  102. 102. • invocation Count • error Count • latency • throttling • granular to the minute
  103. 103. • invocation Count • error Count • latency • throttling • granular to the minute • support custom metrics
  104. 104. • AWS integration • support custom metrics https://www.datadoghq.com/blog/monitoring-lambda-functions-datadog/
  105. 105. Monitor CI/CD Code Test Deploy {}
  106. 106. “Netflix for sports” offices in London, Leeds, Katowice and Tokyo We’re hiring! Visit engineering.dazn.com to learn more. follow @dazneng for updates about the engineering team.
  107. 107. API Gateway and Kinesis Authentication & authorisation (IAM, Cognito) Testing Running & Debugging functions locally Log aggregation Monitoring & Alerting X-Ray Correlation IDs CI/CD Performance and Cost optimisation Error Handling Configuration management VPC Security Leading practices (API Gateway, Kinesis, Lambda) Canary deployments http://bit.ly/production-ready-serverless get 40% off with: ytcui
  108. 108. @theburningmonk theburningmonk.com github.com/theburningmonk

×