1. with

2. Yan Cui
http://theburningmonk.com
@theburningmonk
Principal Engineer @

3. “Netflix for sports”
offices in London, Leeds, Katowice and Tokyo

6. available on 30+ platforms

7. ~500,000 concurrent viewers

8. available in Austria, Switzerland,
Germany, Japan, Canada and Italy
USA coming soon ;-)

11. “Netflix for sports”
offices in London, Leeds, Katowice and Tokyo
We’re hiring! Visit
engineering.dazn.com to
learn more.
follow @dazneng for updates
about the engineering team.

13. AWS user for 10 years

14. 2009…

19. DevOps!

20. as in.. devs doing ops?

21. let’s rebrand the ops team

22. Time

23. Time
it’s working!!!

24. a set of practices to reduce the time
between committing a change to a system
and the change being placed into normal
production, while ensuring high quality

25. a set of practices to reduce the time
between committing a change to a system
and the change being placed into normal
production, while ensuring high quality

28. DevOps toolchain

29. http://bit.ly/2MNoRaY

30. http://bit.ly/2MNoRaY

31. FOCUS

32. one of the biggest distraction
for tech teams has been the
endless tinkering
of tools

35. simple. does the job. gets out of my way.

36. tools not gadgets

37. http://serverless.com/framework
+

38. Code Test Deploy
{}

39. Code Test Deploy
{}

40. github
repo
github
repo
github
repo
github
repo
user-api
timeline-api
relationship-api
search-api

41. Code Test Deploy
{}
infrastructure-as-code

46. Code Test Deploy
{}
infrastructure-as-code
CloudFormation

47. Code Test Deploy
{}

48. https://amzn.to/29Lxuzu

49. Level of Testing
1.Unit
do our objects do the right thing?
are they easy to work with?
https://amzn.to/29Lxuzu

51. Level of Testing
1.Unit
2.Integration
does our code work against code we
can’t change?
https://amzn.to/29Lxuzu

52. handler

53. handler
test by invoking
the handler

54. Level of Testing
1.Unit
2.Integration
3.Acceptance
does the whole system work?
https://amzn.to/29Lxuzu

55. Level of Testing
unit
integration
acceptance
feedback
confidence

56. “…We find that tests that mock external
libraries often need to be complex to
get the code into the right state for the
functionality we need to exercise.
The mess in such tests is telling us that
the design isn’t right but, instead of
fixing the problem by improving the
code, we have to carry the extra
complexity in both code and test…”
Don’t Mock Types You Can’t Change
https://amzn.to/29Lxuzu

57. “…The second risk is that we have to be
sure that the behaviour we stub or mock
matches what the external library will
actually do…
Even if we get it right once, we have to
make sure that the tests remain valid
when we upgrade the libraries…”
Don’t Mock Types You Can’t Change
https://amzn.to/29Lxuzu

58. Don’t Mock Types You Can’t Change
Services
https://amzn.to/29Lxuzu

59. Paul Johnston
The serverless approach to
testing is different and may
actually be easier.
http://bit.ly/2t5viwK

60. LambdaAPI Gateway DynamoDB

61. LambdaAPI Gateway DynamoDB
Unit Tests

62. LambdaAPI Gateway DynamoDB
Unit Tests
Mock/Stub

63. is our request correct?
is the request mapping
set up correctly?is the API resources
configured correctly?
are we assuming the
correct schema?
LambdaAPI Gateway DynamoDB
is Lambda proxy
configured correctly?
is IAM policy set
up correctly?
is the table created?
what unit tests will not tell you…

65. most Lambda functions are simple
have single purpose, the risk of
shipping broken software has largely
shifted to how they integrate with
external services
observation

67. optimize towards shipping working
software, even if it means slowing
down your feedback loop…

68. …if a service can’t provide
you with a relatively easy
way to test the interface in
reality, then you should
consider using another one.
Paul Johnston

69. “…Wherever possible, an acceptance
test should exercise the system end-to-
end without directly calling its internal
code.
An end-to-end test interacts with the
system only from the outside: through
its interface…”
Testing End-to-End

70. Legacy Monolith Amazon Kinesis Amazon Lambda
Amazon CloudSearchAmazon API Gateway Amazon Lambda

71. Legacy Monolith Amazon Kinesis Amazon Lambda
Amazon CloudSearchAmazon API Gateway Amazon Lambda
Test Input

72. Legacy Monolith Amazon Kinesis Amazon Lambda
Amazon CloudSearchAmazon API Gateway Amazon Lambda
Test Input
Validate

73. integration tests exercise
system’s Integration with its
external dependencies
my code

74. acceptance tests exercise
system End-to-End from
the outside
my code

75. integration tests differ from
acceptance tests only in HOW the
Lambda functions are invoked
observation

78. Code Test Deploy
{}

79. Code Test Deploy
{}

80. serverless.yml
{}
Code

81. {}
Code
serverless.yml

82. serverless.yml
{}
Code
S3

83. {}
Code
serverless.yml
S3 CloudFormation

84. {}
Code
serverless.yml
S3 CloudFormation

85. request
blue-green deployment

86. request
blue-green deployment

87. request
blue-green deployment

88. request
blue-green deployment
req/s
auto-scaling

89. request
blue-green deployment
req/s
auto-scaling
us-east-1a
us-east-1b
us-east-1c
multi-AZ

90. request
blue-green deployment
req/s
auto-scaling
us-east-1a
us-east-1b
us-east-1c
multi-AZ

91. Code Test Deploy
{}

93. simple. does the job. gets out of my way.

94. https://amzn.to/2wb9kbt
multi-region active-active

95. canary deployment
http://bit.ly/2OXd6fn

96. Code Test Deploy
{}

97. CI/CD Monitor

98. CI/CD Monitor
drone circleci

103. if [ "$1" = "deploy" ] && [ $# -eq 4 ]; then
STAGE=$2
REGION=$3
PROFILE=$4
npm install
AWS_PROFILE=$PROFILE 'node_modules/.bin/sls' deploy -s $STAGE -r $REGION
elif [ "$1" = "int-test" ] && [ $# -eq 4 ]; then
STAGE=$2
REGION=$3
PROFILE=$4
npm install
AWS_PROFILE=$PROFILE npm run int-$STAGE
elif [ "$1" = "acceptance-test" ] && [ $# -eq 4 ]; then
STAGE=$2
REGION=$3
PROFILE=$4
npm install
AWS_PROFILE=$PROFILE npm run acceptance-$STAGE
else
usage
exit 1
fi

104. if [ "$1" = "deploy" ] && [ $# -eq 4 ]; then
STAGE=$2
REGION=$3
PROFILE=$4
npm install
AWS_PROFILE=$PROFILE 'node_modules/.bin/sls' deploy -s $STAGE -r $REGION
elif [ "$1" = "int-test" ] && [ $# -eq 4 ]; then
STAGE=$2
REGION=$3
PROFILE=$4
npm install
AWS_PROFILE=$PROFILE npm run int-$STAGE
elif [ "$1" = "acceptance-test" ] && [ $# -eq 4 ]; then
STAGE=$2
REGION=$3
PROFILE=$4
npm install
AWS_PROFILE=$PROFILE npm run acceptance-$STAGE
else
usage
exit 1
fi
install Serverless framework
as dev dependency

105. if [ "$1" = "deploy" ] && [ $# -eq 4 ]; then
STAGE=$2
REGION=$3
PROFILE=$4
npm install
AWS_PROFILE=$PROFILE 'node_modules/.bin/sls' deploy -s $STAGE -r $REGION
elif [ "$1" = "int-test" ] && [ $# -eq 4 ]; then
STAGE=$2
REGION=$3
PROFILE=$4
npm install
AWS_PROFILE=$PROFILE npm run int-$STAGE
elif [ "$1" = "acceptance-test" ] && [ $# -eq 4 ]; then
STAGE=$2
REGION=$3
PROFILE=$4
npm install
AWS_PROFILE=$PROFILE npm run acceptance-$STAGE
else
usage
exit 1
fi
install Serverless framework
as dev dependency
mitigate version conflicts

106. if [ "$1" = "deploy" ] && [ $# -eq 4 ]; then
STAGE=$2
REGION=$3
PROFILE=$4
npm install
AWS_PROFILE=$PROFILE 'node_modules/.bin/sls' deploy -s $STAGE -r $REGION
elif [ "$1" = "int-test" ] && [ $# -eq 4 ]; then
STAGE=$2
REGION=$3
PROFILE=$4
npm install
AWS_PROFILE=$PROFILE npm run int-$STAGE
elif [ "$1" = "acceptance-test" ] && [ $# -eq 4 ]; then
STAGE=$2
REGION=$3
PROFILE=$4
npm install
AWS_PROFILE=$PROFILE npm run acceptance-$STAGE
else
usage
exit 1
fi
build.sh allows repeatable builds on both local & CI

108. Auto Auto Manual

110. CI/CD Monitor

111. CI/CD Monitor
logging

113. 2016-07-12T12:24:37.571Z 994f18f9-482b-11e6-8668-53e4eab441ae
GOT is off air, what do I do now?

114. 2016-07-12T12:24:37.571Z 994f18f9-482b-11e6-8668-53e4eab441ae
GOT is off air, what do I do now?
UTC Timestamp API Gateway Request Id
your log message

115. Me
Logs are not easily searchable
in CloudWatch Logs.

116. CloudWatch Logs

117. AWS Lambda
invokes
AWS Lambda
stdout
asynchronously
any log aggregation service
CloudWatch Logs

118. CloudWatch Logs
AWS Lambda
AWS Lambda
stdout
any log aggregation service
asynchronously
invokes

119. …

120. CloudWatch Events

123. CI/CD Monitor
logging tracing

125. a user
my followers didn’t receive
my new post!

126. where could the
problem be?

127. correlation IDs*
* eg. request-id, user-id, yubl-id, etc.

128. wrap HTTP client & AWS SDK clients
to forward captured correlation IDs

129. kinesis client
http client
sns client

130. use X-Ray for performance tracing

131. Amazon X-Ray

132. Amazon X-Ray

133. CI/CD Monitor
logging tracing metrics

134. no place to install agents/daemons

135. • invocation Count
• error Count
• latency
• throttling
• granular to the minute

136. • invocation Count
• error Count
• latency
• throttling
• granular to the minute
• support custom metrics

137. • AWS integration
• support custom metrics
https://www.datadoghq.com/blog/monitoring-lambda-functions-datadog/

144. Monitor CI/CD
Code Test Deploy
{}

145. “Netflix for sports”
offices in London, Leeds, Katowice and Tokyo
We’re hiring! Visit
engineering.dazn.com to
learn more.
follow @dazneng for
updates about the
engineering team.

146. API Gateway and Kinesis
Authentication & authorisation (IAM, Cognito)
Testing
Running & Debugging functions locally
Log aggregation
Monitoring & Alerting
X-Ray
Correlation IDs
CI/CD
Performance and Cost optimisation
Error Handling
Configuration management
VPC
Security
Leading practices (API Gateway, Kinesis, Lambda)
Canary deployments
http://bit.ly/production-ready-serverless
get 40% off
with: ytcui

147. @theburningmonk
theburningmonk.com
github.com/theburningmonk