1.
MIND
THE
POTHOLES
MIND
THE
POTHOLES

2.
What do you mean
by ‘serverless’?

3.
“Serverless”

4.
Gojko Adzic
It is serverless the same way
WiFi is wireless.
http://bit.ly/2yQgwwb

5.
Serverless means…
don’t pay for it if no-one uses it
don’t need to worry about scaling
don’t need to provision and manage servers

6.
in other words, it’s a lot like taking a cab

7.
Ownership
Fuel
Navigate
To get there!
Focus on
getting there!

8.
HW Ownership
OS
Runtime & Scale
Code
Focus on
getting there!
Physical
Servers
Virtual
Machines
Containers Serverless

9.
Nano Services Self Managed Cost Paradigm
ChangeAsync
Dynamic agile env

10.
“why are we failing at this?”

11.
hidden dangers

12.
monolith microservices serverless

13.
monolith microservices serverless
observability
distributed
systems
bounded
context

14.
monolith microservices serverless
observability
distributed
systems
bounded
context

15.
monolith microservices serverless
observability
distributed
systems
bounded
context
event driven

16.
monolith serverless
missing learnings
from microservices

17.
monolith serverless
missing learnings
from microservices
poor decisions

18.
Yan Cui
http://theburningmonk.com
@theburningmonk
AWS user for 10 years

19.
http://bit.ly/yubl-serverless

20.
Yan Cui
http://theburningmonk.com
@theburningmonk
Developer Advocate @

21.
Yan Cui
http://theburningmonk.com
@theburningmonk
Independent Consultant
advisetraining delivery

22.
https://theburningmonk.com/workshops
Amsterdam, March 19-20 Helsinki, May 4-5 Stockholm, May 14-15
Dublin, June 16-17 London, September 24-25
early bird
until Feb 11

23.
#1
not letting go of legacy
thinking

24.
“we’re doing serverless,
but why aren’t thing
going faster?”

25.
Socio Technical

26.
there are no silver bullets

27.
centralised team
Team A Team B Team C Team D …

28.
“but the developers don’t understand AWS and how
our infrastructure is set up”

29.
“but the developers don’t understand AWS and how
our infrastructure is set up”
let’s solve this
problem instead!

30.
what got you here won’t get you there

31.
if (path == “/user” && method == “GET”) {
return getUser(…);
} else if (path == “/user” && method == “DELETE”) {
return deleteUser(…);
} else if (path == “/user” && method == “POST”) {
return createUser(…);
} else if ….
Monolithic Functions

32.
GET /user
POST /user
DELETE /user
Single-Purposed Functions

33.
author: yan.cui
feature: user-api
user-api-dev
Monolithic Single-Purposed
author: yan.cui
feature: user-api
user-api-dev-get-user
author: yan.cui
feature: user-api
user-api-dev-create-user
author: yan.cui
feature: user-api
user-api-dev-delete-user

34.
author: yan.cui
feature: user-api
user-api-dev
Monolithic Single-Purposed
author: yan.cui
feature: user-api
user-api-dev-get-user
author: yan.cui
feature: user-api
user-api-dev-create-user
author: yan.cui
feature: user-api
user-api-dev-delete-user
find related
functions by prefix

35.
author: yan.cui
feature: user-api
user-api-dev
Monolithic Single-Purposed
author: yan.cui
feature: user-api
user-api-dev-get-user
author: yan.cui
feature: user-api
user-api-dev-create-user
author: yan.cui
feature: user-api
user-api-dev-delete-user
discoverability
(without having to dig into the code)

36.
author: yan.cui
feature: user-api
user-api-dev
Monolithic Single-Purposed
author: yan.cui
feature: user-api
user-api-dev-get-user
author: yan.cui
feature: user-api
user-api-dev-create-user
author: yan.cui
feature: user-api
user-api-dev-delete-user
what does it do?

37.
author: yan.cui
feature: user-api
user-api-dev
Monolithic Single-Purposed
author: yan.cui
feature: user-api
user-api-dev-get-user
author: yan.cui
feature: user-api
user-api-dev-create-user
author: yan.cui
feature: user-api
user-api-dev-delete-user
dynamodb:GetItem
dynamodb:PutItem
dynamodb:DeleteItem

38.
author: yan.cui
feature: user-api
user-api-dev
Monolithic Single-Purposed
author: yan.cui
feature: user-api
user-api-dev-get-user
author: yan.cui
feature: user-api
user-api-dev-create-user
author: yan.cui
feature: user-api
user-api-dev-delete-user
dynamodb:GetItem
dynamodb:PutItem
dynamodb:DeleteItem
no least privilege…

39.
author: yan.cui
feature: user-api
user-api-dev
Monolithic Single-Purposed
author: yan.cui
feature: user-api
user-api-dev-get-user
author: yan.cui
feature: user-api
user-api-dev-create-user
author: yan.cui
feature: user-api
user-api-dev-delete-user
require(x)
require(y)
require(z)

40.
author: yan.cui
feature: user-api
user-api-dev
Monolithic Single-Purposed
author: yan.cui
feature: user-api
user-api-dev-get-user
author: yan.cui
feature: user-api
user-api-dev-create-user
author: yan.cui
feature: user-api
user-api-dev-delete-user
require(x)
require(y)
require(z)

41.
more dependecies equals
slower cold start

42.
author: yan.cui
feature: user-api
user-api-dev
Monolithic Single-Purposed
author: yan.cui
feature: user-api
user-api-dev-get-user
author: yan.cui
feature: user-api
user-api-dev-create-user
author: yan.cui
feature: user-api
user-api-dev-delete-user
require(x)
require(y)
require(z)
worse cold start
performance

43.
keep functions simple, and single-purposed

44.
#2
one account that rules
them all

45.
mind the shared limits

46.
no. of DynamoDB tables
no. of API Gateway regional APIs
no. of API Gateway edge-optimized APIs
no. of Kinesis shards
no. of IAM roles
no. of S3 buckets
no. of CloudFormation stacks
no. of SNS subscription filters
no. of SSM parameters
…
Resource Limits

47.
DynamoDB read & write
API Gateway requests/second
Lambda concurrent executions
SSM parameter ops/second
…
Throughput Limits

48.
compartmentalise security breaches

49.
One account per Team per Environment

50.
#3
do first, research later

51.
https://einaregilsson.com/serverless-15-percent-slower-and-eight-times-more-expensive/

52.
the platforms need to do better at educating users on
how to choose between different services

53.
SNS vs SQS vs Kinesis vs MKS?
the platforms need to do better at educating users on
how to choose between different services

54.
ordering
replay events
Kinesis SQS SNS
by shard
none (standard)
global (FIFO)
none
up to 7 days none none
mode
retry
batched batched (up to 10) singular
retried until
success
(customizable)
retry + DLQ retry + DLQ
concurrency 1 per shard auto-scaled fan-out!!!
subscribers many one-to-one many
EventBridge
many
none
none
singular
retry + DLQ
fan-out!!!

55.
https://medium.com/theburningmonk-com/all-my-posts-on-serverless-aws-lambda-43c17a147f91

56.
https://www.jeremydaly.com/newsletter/

57.
#4
not using a deployment
toolkit

58.
https://lumigo.io/blog/comparison-of-lambda-deployment-frameworks/

59.
don’t write your own deployment framework

60.
#5
console-driven
development

61.
#6
one repo per function

62.
github
repo
github
repo
github
repo
github
repo
github
repo
github
repo
github
repo
github
repo
github
repo

63.
github
repo
github
repo
github
repo
github
repo
github
repo
github
repo
github
repo
github
repo
github
repo

64.
monorepo?

65.
github
repo

66.
one repo per service?

67.
github
repo
github
repo
github
repo
github
repo
user-api
timeline-api
relationship-api
search-api

68.
https://lumigo.io/blog/mono-repo-vs-one-per-service/

69.
github
repo
github
repo
github
repo
github
repo
user-api
timeline-api
relationship-api
search-api

70.
CI/CD pipeline per service

71.
functions are deployed together, as a stack

72.
unencrypted secrets
in env vars
#7

73.
secrets should NEVER be in plain text in env variables

74.
SSM Parameter Store
Secret 1
Secret 2
IAM
Environment:
SECRET_1: …
SECRET_2: …
Environment:
SECRET_1: …
SECRET_2: …

75.
SSM Parameter Store
Secret 1
Secret 2
IAM
Environment:
SECRET_1: …
SECRET_2: …
Environment:
SECRET_1: …
SECRET_2: …
yay!

76.
SSM Parameter Store
Secret 1
Secret 2
IAM
fetch at cold start,
cache,
invalidate every x mins

77.
https://github.com/middyjs/middy

78.
SSM Parameter Store
Secret 1
Secret 2
IAM
switch to Higher
Throughput if you need
more than 40 ops/s

79.
not following least
privilege principle
#8

80.
missing DLQs
#9

81.
async sync
S3
SNS
SES
CloudFormation
CloudWatch Logs
CloudWatch Events
Scheduled Events
CodeCommit
AWS Config
http://amzn.to/2v7Kc3b
Cognito
Alexa
Lex
API Gateway
pulling
DynamoDB Stream
Kinesis Stream
SQS

82.
async sync
S3
SNS
SES
CloudFormation
CloudWatch Logs
CloudWatch Events
Scheduled Events
CodeCommit
AWS Config
http://amzn.to/2vs2lIg
Cognito
Alexa
Lex
API Gateway
pulling
DynamoDB Stream
Kinesis Stream
SQS
Lambda handles retries
(twice, then DLQ)

83.
configure DLQ for async functions so you don’t lose failed events

84.
DLQ Lambda Destinations
payload payload, context(s), and response

85.
DLQ Lambda Destinations
payload payload, context(s), and response
DeadLetterErrors no error metrics!

86.
too much/too little
concurrency
#10

87.
“Lambda generates too much load for the downstream system”

88.
one invocation
per message
SNS
Lambda

89.
Downstream
System
SNS
Lambda

90.
ordering
replay events
Kinesis SQS SNS
by shard
none (standard)
global (FIFO)
none
up to 7 days none none
mode
retry
batched batched (up to 10) singular
retried until
success
(customizable)
retry + DLQ retry + DLQ
concurrency 1 per shard auto-scaled fan-out!!!
subscribers many one-to-one many
EventBridge
many
none
none
singular
retry + DLQ
fan-out!!!

91.
if you want…
maximum
throughput
SNS
precise control
over throughput
Kinesis

92.
if you want…
maximum
throughput
SNS
precise control
over throughput
Kinesis
how quickly it scales out

93.
if you want…
maximum
throughput
SNS
precise control
over throughput
Kinesis
how quickly it scales out
SQS DynamoDB
Streams

94.
ordering
replay events
Kinesis SQS SNS
by shard
none (standard)
global (FIFO)
none
up to 7 days none none
mode
retry
batched batched (up to 10) singular
retried until
success
(customizable)
retry + DLQ retry + DLQ
concurrency 1 per shard auto-scaled fan-out!!!
subscribers many one-to-one many
EventBridge
many
none
none
singular
retry + DLQ
fan-out!!!

95.
cold starts
#11

96.
“cold starts only happen to the first request”

97.
function invocationconcurrent execution
i.e. a container

98.
function invocationconcurrent execution
i.e. a container
class instance method call

99.
Lambda scales the number of concurrent executions
based on traffic

100.
existing “containers” are reused where possible

101.
time
invocation

102.
time
invocation
invocation

103.
time
invocation
invocation

104.
time
invocation
invocation
invocation
invocation

105.
time
invocation
invocation
invocation
invocation
invocation
invocation

106.
time
invocation
invocation
invocation
invocation
invocation
invocation

107.
time
invocation
invocation
invocation
invocation
invocation
invocation
invocation

108.
time
invocation
invocation
invocation
invocation
invocation
invocation
invocation invocation

109.
time
invocation
invocation
invocation
invocation
invocation
invocation
invocation invocation

110.
time
invocation
invocation
invocation
invocation
invocation
invocation
invocation invocation

111.
time
invocation
invocation
ping
invocation
invocation
invocation
ping ping

112.
Lambda warmers don’t work when you have > 1
concurrent executions

113.
FREQUENCY DURATION

114.
FREQUENCY DURATION
dictated by user traffic,
out of your control

115.
cold starts is generally not an issue if you have a
steady traffic pattern

116.
time
req/s

117.
time
req/s
El Classico

118.
time
req/s
lunch dinner

119.
FREQUENCY DURATION
optimize this!

120.
minimise the duration of cold starts so they
fall within acceptable latency range

121.
time
req/s
lunch dinner
Provisioned
Concurrency

122.
time
req/s
lunch dinner
Provisioned
Concurrency
On-Demand
Concurrency

123.
https://lumigo.io/blog/provisioned-concurrency-the-end-of-cold-starts/

124.
there are no silver bullets

125.
reserved concurrency is a powerful tool IFF you
have a cold start problem
don’t use it by default

126.
RDS connection
handling
#12

127.
default RDS configs are bad for Lambda

128.
default RDS configs are bad for Lambda
idle connections are
not closed
too many connections
per “container”
max open connection
is too low

129.
https://www.jeremydaly.com/manage-rds-connections-aws-lambda/

130.
set “wait_timeout” and “interactive_timeout” to 10 mins
(default is 8 hours!)

131.
increase “max_connections” setting

132.
set client socket pool size to 1

133.
(lack of) observability
#13

134.
happened system repaireduser impact
reduce MTTR

135.
Identify & Resolve
Issues
Understanding
costs
Visibility

136.
Identify & Resolve
Issues
Understanding
costs
Visibility

137.
happened system repaireduser impact
MTTDiscovery

138.
“What alerts should I have?”

139.
It depends on what you’re building…

140.
But, this is a good starting point

141.
Lambda
error rate %
throttle count
DLR error count
iterator age
regional concurrency

142.
Lambda
error rate %
throttle count
DLR error count
iterator age
regional concurrency
API Gateway
p90/95/99 latency
success rate %
4xx rate %
5xx rate %

143.
API Gateway
p90/95/99 latency
success rate %
4xx rate %
5xx rate %
SQS
message age
Lambda
error rate %
throttle count
DLR error count
iterator age
regional concurrency

144.
API Gateway
p90/95/99 latency
success rate %
4xx rate %
5xx rate %
SQS
message age
Step Functions
failed count
throttle count
timed out count
Lambda
error rate %
throttle count
DLR error count
iterator age
regional concurrency

145.
SQS
message age
Step Functions
failed count
throttle count
timed out count
API Gateway
p90/95/99 latency
success rate %
4xx rate %
5xx rate %
Lambda
error rate %
throttle count
DLR error count
iterator age
regional concurrency

146.
monitor and alert on message flow rate for
event processing pipelines

147.
“Can’t you codify these?”

148.
https://theburningmonk.com/hire-me
AdviseTraining Delivery
“Fundamentally, Yan has improved our team by increasing our
ability to derive value from AWS and Lambda in particular.”
Nick Blair
Tech Lead

149.
Production-Ready Serverless

150.
bit.ly/prod-ready-sls-ams-2019
Backspace Escape Rooms
Transformatorweg 30A
20% off with
aws-ug-ams-0109

151.
https://theburningmonk.com/workshops
Amsterdam, March 19-20 Helsinki, May 4-5 Stockholm, May 14-15
Dublin, June 16-17 London, September 24-25
early bird
until Feb 11

152.
@theburningmonk
theburningmonk.com
github.com/theburningmonk