1. How to build a social
network on #serverless
Yan Cui
@theburningmonk

2. Yan Cui
http://theburningmonk.com
@theburningmonk
Principal Engineer @
Independent Consultant
Instructor @
Instructor @
Advisor @

3. “Netflix for sports”
offices in London, Leeds, Katowice and Amsterdam

5. available in Austria, Switzerland, Germany,
Japan, Italy, Spain, Canada, USA, and Brazil
available on 30+ platforms

6. 1,000,000+ concurrent viewers

7. “Netflix for sports”
offices in London, Leeds, Katowice and Amsterdam
We’re hiring! Visit
engineering.dazn.com to
learn more.
follow @dazneng for updates
about the engineering team.
WE’RE HIRING!

10. apr, 2016

11. nov, 2016

12. WHY?

14. hey guys, vote on this post
and I’ll announce a winner at
10PM tonight

15. 10PM
traffic

16. 10PM
traffic
70-100x

17. low utilisation to leave room for spikes
EC2 scaling is slow, so scale earlier

18. lots of $$$ for unused resources

19. up to 30 mins for deployment
deployment required downtime

20. features took months to develop

21. - Dan North
“lead time to someone saying
thank you is the only reputation
metric that matters.”

23. WHY?
to deliver better UX

24. WHY?
to deliver better UX
to deliver value faster

25. WHY?
to deliver better UX
to deliver value faster
to be more cost efficient

26. WHY?
to deliver better UX
to deliver value faster
to be more cost efficient
HOW?

27. what would good look like for us?

28. small
fast
zero downtime
no lock-step
deployments should be…

29. features should be…
deployable independently
loosely-coupled

30. we want to…
minimise cost for unused resources

31. we want to…
minimise cost for unused resources
minimise ops effort

32. we want to…
minimise cost for unused resources
minimise ops effort
reduce tech mess

33. we want to…
minimise cost for unused resources
minimise ops effort
reduce tech mess
deliver visible improvements faster

34. WHY?
to deliver better UX
to deliver value faster
to be more cost efficient
HOW?
microservices

35. WHY?
to deliver better UX
to deliver value faster
to be more cost efficient
HOW?
microservices
event-driven

36. WHY?
to deliver better UX
to deliver value faster
to be more cost efficient
HOW?
microservices
event-driven
serverless

37. WHY?
to deliver better UX
to deliver value faster
to be more cost efficient
HOW?
microservices
event-driven
serverless
WHAT?
this talk!

38. WHY?
to deliver better UX
to deliver value faster
to be more cost efficient
HOW?
microservices
event-driven
serverless
WHAT?
this talk!

39. 170 Lambda functions in prod

40. 95% cost saving vs. EC2

41. 15x no. of prod releases per month

42. 15x no. of prod releases per month
(features were sometimes implemented on the same day)

43. time
is a good fit

44. 1st function in prod!
time
is a good fit

45. ?
time
is a good fit
1st function in prod!

46. CI/CD?

47. CI/CD?
testing?

48. CI/CD?
testing?
logging, monitoring, alerting?

49. time
is a good fit
1st function in prod!
CI/CD, testing, logging,
monitoring, alerting

50. 170 functions
?
time
is a good fit
1st function in prod!
CI/CD, testing, logging,
monitoring, alerting

51. tracing?

52. tracing?
config management?

53. tracing?
config management?
security?

54. 170 functions
time
is a good fit
1st function in prod!
CI/CD, testing, logging,
monitoring, alerting
tracing, config
management, security

55. API Gateway and Kinesis
Authentication & authorisation (IAM, Cognito)
Testing
Running & Debugging functions locally
Log aggregation
Monitoring & Alerting
X-Ray
Correlation IDs
CI/CD
Performance and Cost optimisation
Error Handling
Configuration management
VPC
Security
Leading practices (API Gateway, Kinesis, Lambda)
Canary deployments
http://bit.ly/production-ready-serverless
get 40% off
with: ytcui

56. evolving the PLATFORM

57. Legacy Monolith Amazon Kinesis
Step 1.
ALL state changes!

58. events are an enabler for
COMPOSABILITY

59. AWS LAMBDA
is the...

60. Kinesis

61. Kinesis
API Gateway AWS Lambda API GatewayAWS Lambda
service-A service-B

62. Kinesis
API Gateway AWS Lambda API GatewayAWS Lambda
service-A service-B

63. Kinesis
API Gateway AWS Lambda API GatewayAWS Lambda
service-A service-B
AWS Lambda
AWS Lambda
AWS Lambda

64. Kinesis
API Gateway AWS Lambda API GatewayAWS Lambda
service-A service-B
AWS Lambda
AWS Lambda
AWS Lambda DynamoDBIOT

65. Kinesis
API Gateway AWS Lambda API GatewayAWS Lambda
service-A service-B
AWS Lambda
AWS Lambda
AWS Lambda DynamoDBIOT

66. Kinesis
API Gateway AWS Lambda API GatewayAWS Lambda
service-A service-B
AWS Lambda
AWS Lambda
AWS Lambda DynamoDBIOT
AWS Lambda AWS Lambda

67. build loosely-coupled system
through events

68. service A service B
service C service D
bounded context
bounded context

69. service A service B
service C service D
bounded context
bounded context

70. service A service B
service C service D

72. there are no silver bullets

73. service A service B
service C service D

74. service A service B
service C service D

75. service A service B
service C service D
update!

76. service A service B
service C service Dbackward-compatible?
update!

78. bounded context
DON’T use events to
orchestrate workflows
within the same
bounded context

79. bounded context
adds unnecessary
complexity to logging,
tracing, and end-to-end
reporting

80. bounded context
the workflow doesn’t exist
as a standalone concept,
but as the sum of a series of
loosely connected parts

81. Step Functions
use Step Functions instead

82. Step Functions
don’t forget to emit events
from the workflow

83. Step Functions
so others can react to state
changes that happened as
part of the workflow

84. “how do I organize my functions
into code repositories?”

85. monorepo?

86. github
repo

87. https://lumigo.io/blog/mono-repo-vs-one-per-service/

88. monorepo !== monostack

89. one repo per service?

97. github
repo
github
repo
github
repo
github
repo
user-api
timeline-api
relationship-api
search-api

98. CI/CD pipeline per service

99. functions are deployed
together, as a stack

100. Strangler Pattern
incrementally migrate the legacy system by
gradually replacing pieces of functionalities
to the new system

102. rebuilt search

103. Legacy Monolith Amazon Kinesis Amazon Lambda
Amazon CloudSearch

104. Legacy Monolith Amazon Kinesis Amazon Lambda
Amazon CloudSearchAmazon API Gateway Amazon Lambda

105. proxy requests from monolith
to new service

106. new analytics pipeline

107. expensive ($3000/month)
don’t understand our domain
JS based query language

108. Legacy Monolith Amazon Kinesis Amazon Lambda
Google BigQuery

109. Legacy Monolith Amazon Kinesis Amazon Lambda
Google BigQuery
1 developer, 2 days
design production
(his 1st serverless project)

110. Legacy Monolith Amazon Kinesis Amazon Lambda
Google BigQuery
“nothing ever got done
this fast at Skype!”
- Chris Twamley

111. - Dan North
“lead time to someone saying
thank you is the only reputation
metric that matters.”

112. $3000/month $0.03/month

114. Kinesis
sink

115. Kinesis Kinesis Firehose
batch Kinesis events

116. Kinesis Kinesis Firehose S3
data lake

117. Kinesis Kinesis Firehose S3
Glue
analyze data schema,
catalog data into tables

118. Kinesis Kinesis Firehose S3
Athena
Glue
query engine

119. Kinesis Kinesis Firehose S3
AthenaQuickSight
Glue
visualization, dashboards

120. Kinesis Kinesis Firehose S3
AthenaQuickSight
Glue
no code is required!

121. Kinesis Kinesis Firehose S3
AthenaQuickSight
Glue
no code is required!
pay-per-use!

122. user action business intelligence

123. user action business intelligence

126. Problem
didn’t work…

127. Problem
didn’t work…
over-engineered…

129. try figure out what’s
going on here…

130. Problem
didn’t work…
over-engineered…
didn’t scale…

132. Rebuilt
with Lambda

138. built-in retry
and DLQ

139. built-in retry
and DLQ
avoid repeating expensive
work of fetching mils of
relationships

143. github
repo
timeline-api
service: timeline-api
provider:
name: aws
runtime: nodejs6.10
stage: dev
region: us-east-1
functions:
distribute-yubl:
…
undistribute-yubl:
…

145. Problem
didn’t work…

146. “it returns the first 30 users in the
database, by creation time…”

148. Rebuilt
with Lambda

149. BigQuery

150. BigQuery

151. grapheneDB
BigQuery

152. grapheneDB
BigQuery

153. grapheneDB
BigQuery

154. grapheneDB
BigQuery
mostly built in one sleepless night…

155. Building a
scalable
notification
system

156. expensive ($3000/month)
don’t understand our domain
JS based query language

157. all the analytics data is
already in BigQuery
powerful query engine

158. all the analytics data is
already in BigQuery
powerful query engine

159. Design Goals
ad-hoc notifications

160. Design Goals
ad-hoc notifications
scheduled notifications

161. Design Goals
ad-hoc notifications
scheduled notifications
A/B testing

163. Design Goals
ad-hoc notifications
scheduled notifications
A/B testing
scalable

164. Design Goals
ad-hoc notifications
scheduled notifications
A/B testing
scalable
cost-effective

165. scheduled notifications

168. how to send notifications
what to send

169. other processes can leverage
this capability of sending
notifications

170. why not SNS?

171. ad-hoc notifications

172. Oversight vs. Frictionless

173. Oversight vs. Frictionless
don’t make life difficult for
the marketing team

174. Oversight vs. Frictionless
don’t make life difficult for
the marketing team
don’t let marketing
team spam users

175. Oversight vs. Frictionless
don’t make life difficult for
the marketing team
don’t let marketing
team spam users
driving usage/engagement
maintaining user experience

176. Marketing work with BI on query
request approval from CPO/CTO
approver checks impact and tests
message format
send notifications

188. more Scalable
(and scales faster!)

189. Cheaper
(don’t pay for idle servers)

190. Resilience
(built-in redundancy and multi-AZ)

191. Secure

192. request
blue-green deployment
req/s
auto-scaling
us-east-1a
us-east-1b
us-east-1c
multi-AZ

193. idea production
greater Velocity from idea to product

194. WHY?
to deliver better UX
to deliver value faster
to be more cost efficient

195. @theburningmonk
theburningmonk.com
github.com/theburningmonk