This document discusses building a social network using serverless architecture. It describes how the company moved from monolithic architecture to microservices, events, and serverless functions. This reduced costs by 95% compared to EC2 and allowed 15x more production releases per month. It also discusses challenges of testing, monitoring, security and other aspects of building serverless systems at scale.

1. How to build a social
network on #serverless
Yan Cui
@theburningmonk

2. Yan Cui
http://theburningmonk.com
@theburningmonk
Principal Engineer @
Independent Consultant
Instructor @
Instructor @
Advisor @

3. “Netflix for sports”
offices in London, Leeds, Katowice and Amsterdam

7. available in Austria, Switzerland, Germany,
Japan, Italy, Spain, Canada and USA
available on 30+ platforms

8. ~1,000,000 concurrent viewers

9. “Netflix for sports”
offices in London, Leeds, Katowice and Amsterdam
We’re hiring! Visit
engineering.dazn.com to
learn more.
follow @dazneng for updates
about the engineering team.

12. apr, 2016

13. nov, 2016

14. WHY?

16. hey guys, vote on this post
and I’ll announce a winner at
10PM tonight

17. 10PM
traffic

18. 10PM
traffic
70-100x

19. low utilisation to leave room for spikes
EC2 scaling is slow, so scale earlier

20. lots of $$$ for unused resources

21. up to 30 mins for deployment
deployment required downtime

22. features took months to develop

23. - Dan North
“lead time to someone saying
thank you is the only reputation
metric that matters.”

25. WHY?
to deliver better UX

26. WHY?
to deliver better UX
to deliver value faster

27. WHY?
to deliver better UX
to deliver value faster
to be more cost efficient

28. WHY?
to deliver better UX
to deliver value faster
to be more cost efficient
HOW?

29. what would good look like for us?

30. small
fast
zero downtime
no lock-step
deployments should be…

31. features should be…
deployable independently
loosely-coupled

32. we want to…
minimise cost for unused resources

33. we want to…
minimise cost for unused resources
minimise ops effort

34. we want to…
minimise cost for unused resources
minimise ops effort
reduce tech mess

35. we want to…
minimise cost for unused resources
minimise ops effort
reduce tech mess
deliver visible improvements faster

36. WHY?
to deliver better UX
to deliver value faster
to be more cost efficient
HOW?
microservices

37. WHY?
to deliver better UX
to deliver value faster
to be more cost efficient
HOW?
microservices
event-driven

38. WHY?
to deliver better UX
to deliver value faster
to be more cost efficient
HOW?
microservices
event-driven
serverless

39. WHY?
to deliver better UX
to deliver value faster
to be more cost efficient
HOW?
microservices
event-driven
serverless
WHAT?
this talk!

40. WHY?
to deliver better UX
to deliver value faster
to be more cost efficient
HOW?
microservices
event-driven
serverless
WHAT?
this talk!

41. 170 Lambda functions in prod

42. 95% cost saving vs. EC2

43. 15x no. of prod releases per month

44. 15x no. of prod releases per month
(features were sometimes implemented on the same day)

45. time
is a good fit

46. 1st function in prod!
time
is a good fit

47. ?
time
is a good fit
1st function in prod!

48. CI/CD?

49. CI/CD?
testing?

50. CI/CD?
testing?
logging, monitoring, alerting?

51. time
is a good fit
1st function in prod!
CI/CD, testing, logging,
monitoring, alerting

52. 170 functions
?
time
is a good fit
1st function in prod!
CI/CD, testing, logging,
monitoring, alerting

53. tracing?

54. tracing?
config management?

55. tracing?
config management?
security?

56. 170 functions
time
is a good fit
1st function in prod!
CI/CD, testing, logging,
monitoring, alerting
tracing, config
management, security

57. API Gateway and Kinesis
Authentication & authorisation (IAM, Cognito)
Testing
Running & Debugging functions locally
Log aggregation
Monitoring & Alerting
X-Ray
Correlation IDs
CI/CD
Performance and Cost optimisation
Error Handling
Configuration management
VPC
Security
Leading practices (API Gateway, Kinesis, Lambda)
Canary deployments
http://bit.ly/production-ready-serverless
get 40% off
with: ytcui

58. evolving the PLATFORM

59. Legacy Monolith Amazon Kinesis
Step 1.
ALL state changes!

60. events are an enabler for
COMPOSABILITY

61. AWS LAMBDA
is the...

62. Kinesis

63. Kinesis
API Gateway AWS Lambda API GatewayAWS Lambda
service-A service-B

64. Kinesis
API Gateway AWS Lambda API GatewayAWS Lambda
service-A service-B

65. Kinesis
API Gateway AWS Lambda API GatewayAWS Lambda
service-A service-B
AWS Lambda
AWS Lambda
AWS Lambda

66. Kinesis
API Gateway AWS Lambda API GatewayAWS Lambda
service-A service-B
AWS Lambda
AWS Lambda
AWS Lambda DynamoDBIOT

67. Kinesis
API Gateway AWS Lambda API GatewayAWS Lambda
service-A service-B
AWS Lambda
AWS Lambda
AWS Lambda DynamoDBIOT

68. Kinesis
API Gateway AWS Lambda API GatewayAWS Lambda
service-A service-B
AWS Lambda
AWS Lambda
AWS Lambda DynamoDBIOT
AWS Lambda AWS Lambda

69. build loosely-coupled system
through events

70. service A service B
service C service D
bounded context
bounded context

71. service A service B
service C service D
bounded context
bounded context

72. service A service B
service C service D

74. there are no silver bullets

75. service A service B
service C service D

76. service A service B
service C service D

77. service A service B
service C service D
update!

78. service A service B
service C service Dbackward-compatible?
update!

80. bounded context
DON’T use events to
orchestrate workflows
within the same
bounded context

81. bounded context
adds unnecessary
complexity to logging,
tracing, and end-to-end
reporting

82. bounded context
the workflow doesn’t exist
as a standalone concept,
but as the sum of a series of
loosely connected parts

83. Step Functions
use Step Functions instead

84. Step Functions
don’t forget to emit events
from the workflow

85. Step Functions
so others can react to state
changes that happened as
part of the workflow

86. “how do I organize my functions
into code repositories?”

87. monorepo?

88. github
repo

89. monorepo !== monostack

90. one repo per service?

98. github
repo
github
repo
github
repo
github
repo
user-api
timeline-api
relationship-api
search-api

99. CI/CD pipeline per service

100. functions are deployed
together, as a stack

101. Strangler Pattern
incrementally migrate the legacy system by
gradually replacing pieces of functionalities
to the new system

103. rebuilt search

104. Legacy Monolith Amazon Kinesis Amazon Lambda
Amazon CloudSearch

105. Legacy Monolith Amazon Kinesis Amazon Lambda
Amazon CloudSearchAmazon API Gateway Amazon Lambda

106. proxy requests from monolith
to new service

107. new analytics pipeline

108. expensive ($3000/month)
don’t understand our domain
JS based query language

109. Legacy Monolith Amazon Kinesis Amazon Lambda
Google BigQuery

110. Legacy Monolith Amazon Kinesis Amazon Lambda
Google BigQuery
1 developer, 2 days
design production
(his 1st serverless project)

111. Legacy Monolith Amazon Kinesis Amazon Lambda
Google BigQuery
“nothing ever got done
this fast at Skype!”
- Chris Twamley

112. - Dan North
“lead time to someone saying
thank you is the only reputation
metric that matters.”

113. $3000/month $0.03/month

115. Kinesis
sink

116. Kinesis Kinesis Firehose
batch Kinesis events

117. Kinesis Kinesis Firehose S3
data lake

118. Kinesis Kinesis Firehose S3
Glue
analyze data schema,
catalog data into tables

119. Kinesis Kinesis Firehose S3
Athena
Glue
query engine

120. Kinesis Kinesis Firehose S3
AthenaQuickSight
Glue
visualization, dashboards

121. Kinesis Kinesis Firehose S3
AthenaQuickSight
Glue
no code is required!

122. Kinesis Kinesis Firehose S3
AthenaQuickSight
Glue
no code is required!
pay-per-use!

123. user action business intelligence

124. user action business intelligence

127. Problem
didn’t work…

128. Problem
didn’t work…
over-engineered…

130. try figure out what’s
going on here…

131. Problem
didn’t work…
over-engineered…
didn’t scale…

133. Rebuilt
with Lambda

139. built-in retry
and DLQ

140. built-in retry
and DLQ
avoid repeating expensive
work of fetching mils of
relationships

144. github
repo
timeline-api
service: timeline-api
provider:
name: aws
runtime: nodejs6.10
stage: dev
region: us-east-1
functions:
distribute-yubl:
…
undistribute-yubl:
…

146. Problem
didn’t work…

147. “it returns the first 30 users in the
database, by creation time…”

149. Rebuilt
with Lambda

150. BigQuery

151. BigQuery

152. grapheneDB
BigQuery

153. grapheneDB
BigQuery

154. grapheneDB
BigQuery

155. grapheneDB
BigQuery
mostly built in one sleepless night…

156. Building a
scalable
notification
system

157. expensive ($3000/month)
don’t understand our domain
JS based query language

158. all the analytics data is
already in BigQuery
powerful query engine

159. all the analytics data is
already in BigQuery
powerful query engine

160. Design Goals
ad-hoc notifications

161. Design Goals
ad-hoc notifications
scheduled notifications

162. Design Goals
ad-hoc notifications
scheduled notifications
A/B testing

164. Design Goals
ad-hoc notifications
scheduled notifications
A/B testing
scalable

165. Design Goals
ad-hoc notifications
scheduled notifications
A/B testing
scalable
cost-effective

166. scheduled notifications

169. how to send notifications
what to send

170. other processes can leverage
this capability of sending
notifications

171. why not SNS?

172. ad-hoc notifications

173. Oversight vs. Frictionless

174. Oversight vs. Frictionless
don’t make life difficult for
the marketing team

175. Oversight vs. Frictionless
don’t make life difficult for
the marketing team
don’t let marketing
team spam users

176. Oversight vs. Frictionless
don’t make life difficult for
the marketing team
don’t let marketing
team spam users
driving usage/engagement
maintaining user experience

177. Marketing work with BI on query
request approval from CPO/CTO
approver checks impact and tests
message format
send notifications

189. @theburningmonk
theburningmonk.com
github.com/theburningmonk