MIND
THE
POTHOLES
MIND
THE
POTHOLES

@theburningmonk theburningmonk.com
back in the day…

we’re getting a new server
in 3 months time

yay!
about time!
hooray!!
finally!

but we have to decide
what dependencies to
install on it now..

…

@theburningmonk theburningmonk.com
on premise VMs in the cloud

@theburningmonk theburningmonk.com
on premise VMs in the cloud
abstraction level

@theburningmonk theburningmonk.com
on premise VMs in the cloud
abstraction levelproductivity

@theburningmonk theburningmonk.com
on premise VMs in the cloud
abstraction levelproductivity

@theburningmonk theburningmonk.com
less is more

@theburningmonk theburningmonk.com

@theburningmonk theburningmonk.com

@theburningmonk theburningmonk.com
right?

@theburningmonk theburningmonk.com

@theburningmonk theburningmonk.com
“why are we failing at this?”

hidden dangers

@theburningmonk theburningmonk.com
monolith microservices serverless

@theburningmonk theburningmonk.com
monolith microservices serverless
observability
distributed
systems
bounded
context

@theburningmonk theburningmonk.com
monolith microservices serverless
observability
distributed
systems
bounded
context

@theburningmonk theburningmonk.com
monolith microservices serverless
observability
distributed
systems
bounded
context
event driven

@theburningmonk theburningmonk.com
monolith serverless
missing learnings
from microservices

@theburningmonk theburningmonk.com
monolith serverless
missing learnings
from microservices
poor decisions

Yan Cui
http://theburningmonk.com
@theburningmonk
AWS user for 10 years

http://bit.ly/yubl-serverless

Yan Cui
http://theburningmonk.com
@theburningmonk
Developer Advocate @

Yan Cui
http://theburningmonk.com
@theburningmonk
Independent Consultant
advisetraining delivery

theburningmonk.com/courses

theburningmonk.com/workshops
in your
company
flexible datesHelsinki, Aug 20-21 London, Sep 24-25 Berlin, Oct 8-9
4-week virtual workshop, May 4 - May 29
Amsterdam, Jul 7-8

@theburningmonk theburningmonk.com
not letting go of legacy thinking
one account that rules them all
do first, research later
not using a deployment framework
console-driven development
one repo per function
unencrypted secrets in env vars
not following least privilege principle
missing DLQs
too much/too little concurrency
cold starts
RDS connection handling
(lack of) observability
#1
#2
#3
#4
#5
#6
#7
#8
#9
#10
#11
#12
#13

#1
not letting go of legacy
thinking

“we’re doing serverless,
but why aren’t thing
going faster?”

@theburningmonk theburningmonk.com
Socio Technical

@theburningmonk theburningmonk.com
there are no silver bullets

@theburningmonk theburningmonk.com

@theburningmonk theburningmonk.com
centralised team
Team A Team B Team C Team D …

@theburningmonk theburningmonk.com
“but the developers don’t understand AWS and how
our infrastructure is set up”

@theburningmonk theburningmonk.com
“but the developers don’t understand AWS and how
our infrastructure is set up”
let’s solve this
problem instead!

@theburningmonk theburningmonk.com
what got you here won’t get you there

@theburningmonk theburningmonk.com
if (path == “/user” && method == “GET”) {
return getUser(…);
} else if (path == “/user” && method == “DELETE”) {
return deleteUser(…);
} else if (path == “/user” && method == “POST”) {
return createUser(…);
} else if ….
Monolithic Functions

@theburningmonk theburningmonk.com
GET /user
POST /user
DELETE /user
Single-Purposed Functions

@theburningmonk theburningmonk.com
author: yan.cui
feature: user-api
user-api-dev
Monolithic Single-Purposed
author: yan.cui
feature: user-api
user-api-dev-get-user
author: yan.cui
feature: user-api
user-api-dev-create-user
author: yan.cui
feature: user-api
user-api-dev-delete-user

@theburningmonk theburningmonk.com
author: yan.cui
feature: user-api
user-api-dev
Monolithic Single-Purposed
author: yan.cui
feature: user-api
user-api-dev-get-user
author: yan.cui
feature: user-api
user-api-dev-create-user
author: yan.cui
feature: user-api
user-api-dev-delete-user
find related
functions by prefix

@theburningmonk theburningmonk.com
author: yan.cui
feature: user-api
user-api-dev
Monolithic Single-Purposed
author: yan.cui
feature: user-api
user-api-dev-get-user
author: yan.cui
feature: user-api
user-api-dev-create-user
author: yan.cui
feature: user-api
user-api-dev-delete-user
discoverability
(without having to dig into the code)

@theburningmonk theburningmonk.com
author: yan.cui
feature: user-api
user-api-dev
Monolithic Single-Purposed
author: yan.cui
feature: user-api
user-api-dev-get-user
author: yan.cui
feature: user-api
user-api-dev-create-user
author: yan.cui
feature: user-api
user-api-dev-delete-user
what does it do?

@theburningmonk theburningmonk.com
author: yan.cui
feature: user-api
user-api-dev
Monolithic Single-Purposed
author: yan.cui
feature: user-api
user-api-dev-get-user
author: yan.cui
feature: user-api
user-api-dev-create-user
author: yan.cui
feature: user-api
user-api-dev-delete-user
dynamodb:GetItem
dynamodb:PutItem
dynamodb:DeleteItem

@theburningmonk theburningmonk.com
author: yan.cui
feature: user-api
user-api-dev
Monolithic Single-Purposed
author: yan.cui
feature: user-api
user-api-dev-get-user
author: yan.cui
feature: user-api
user-api-dev-create-user
author: yan.cui
feature: user-api
user-api-dev-delete-user
dynamodb:GetItem
dynamodb:PutItem
dynamodb:DeleteItem
no least privilege…

@theburningmonk theburningmonk.com
author: yan.cui
feature: user-api
user-api-dev
Monolithic Single-Purposed
author: yan.cui
feature: user-api
user-api-dev-get-user
author: yan.cui
feature: user-api
user-api-dev-create-user
author: yan.cui
feature: user-api
user-api-dev-delete-user
require(x)
require(y)
require(z)

@theburningmonk theburningmonk.com
author: yan.cui
feature: user-api
user-api-dev
Monolithic Single-Purposed
author: yan.cui
feature: user-api
user-api-dev-get-user
author: yan.cui
feature: user-api
user-api-dev-create-user
author: yan.cui
feature: user-api
user-api-dev-delete-user
require(x)
require(y)
require(z)

@theburningmonk theburningmonk.com

@theburningmonk theburningmonk.com
more dependecies equals
slower cold start

@theburningmonk theburningmonk.com
author: yan.cui
feature: user-api
user-api-dev
Monolithic Single-Purposed
author: yan.cui
feature: user-api
user-api-dev-get-user
author: yan.cui
feature: user-api
user-api-dev-create-user
author: yan.cui
feature: user-api
user-api-dev-delete-user
require(x)
require(y)
require(z)
worse cold start
performance

@theburningmonk theburningmonk.com
keep functions simple, and single-purposed

#2
one account that rules
them all

@theburningmonk theburningmonk.com
mind the shared limits

@theburningmonk theburningmonk.com
no. of DynamoDB tables
no. of API Gateway regional APIs
no. of API Gateway edge-optimized APIs
no. of Kinesis shards
no. of IAM roles
no. of S3 buckets
no. of CloudFormation stacks
no. of SNS subscription filters
no. of SSM parameters
…
Resource Limits

@theburningmonk theburningmonk.com
DynamoDB read & write
API Gateway requests/second
Lambda concurrent executions
SSM parameter ops/second
…
Throughput Limits

@theburningmonk theburningmonk.com

@theburningmonk theburningmonk.com
compartmentalise security breaches

@theburningmonk theburningmonk.com
One account per Team per Environment

@theburningmonk theburningmonk.com

@theburningmonk theburningmonk.com
https://github.com/OlafConijn/AwsOrganizationFormation

@theburningmonk theburningmonk.com
https://github.com/OlafConijn/AwsOrganizationFormation

@theburningmonk theburningmonk.com
https://github.com/OlafConijn/AwsOrganizationFormation
Accounts
Org Units
SCPs
Pwd Policies
Multi-Region
Pseudo-Funs
Init & Validate
CI/CD

#3
do first, research later

@theburningmonk theburningmonk.com
https://einaregilsson.com/serverless-15-percent-slower-and-eight-times-more-expensive/

@theburningmonk theburningmonk.com

@theburningmonk theburningmonk.com

@theburningmonk theburningmonk.com

@theburningmonk theburningmonk.com
the platforms need to do better at educating users on
how to choose between different services

@theburningmonk theburningmonk.com
SNS vs SQS vs Kinesis vs MKS?
the platforms need to do better at educating users on
how to choose between different services

@theburningmonk theburningmonk.com
ordering
replay events
Kinesis SQS SNS
by shard
none (standard)
global (FIFO)
none
up to 7 days none none
mode
retry
batched batched (up to 10) singular
retried until
success
(customizable)
retry + DLQ retry + DLQ
concurrency 1 per shard auto-scaled fan-out!!!
subscribers many one-to-one many
EventBridge
many
none
none
singular
retry + DLQ
fan-out!!!

@theburningmonk theburningmonk.com
https://medium.com/theburningmonk-com/all-my-posts-on-serverless-aws-lambda-43c17a147f91

@theburningmonk theburningmonk.com
https://www.jeremydaly.com/newsletter/

#4
not using a deployment
toolkit

@theburningmonk theburningmonk.com

@theburningmonk theburningmonk.com
https://lumigo.io/blog/comparison-of-lambda-deployment-frameworks/

@theburningmonk theburningmonk.com
don’t write your own deployment framework

#5
console-driven
development

@theburningmonk theburningmonk.com

@theburningmonk theburningmonk.com

#6
one repo per function

@theburningmonk theburningmonk.com
github
repo
github
repo
github
repo
github
repo
github
repo
github
repo
github
repo
github
repo
github
repo

@theburningmonk theburningmonk.com
github
repo
github
repo
github
repo
github
repo
github
repo
github
repo
github
repo
github
repo
github
repo

@theburningmonk theburningmonk.com
monorepo?

@theburningmonk theburningmonk.com
github
repo

@theburningmonk theburningmonk.com
one repo per service?

@theburningmonk theburningmonk.com
github
repo
github
repo
github
repo
github
repo
user-api
timeline-api
relationship-api
search-api

@theburningmonk theburningmonk.com
https://lumigo.io/blog/mono-repo-vs-one-per-service

@theburningmonk theburningmonk.com

@theburningmonk theburningmonk.com

@theburningmonk theburningmonk.com

@theburningmonk theburningmonk.com

@theburningmonk theburningmonk.com

@theburningmonk theburningmonk.com

@theburningmonk theburningmonk.com

@theburningmonk theburningmonk.com
github
repo
github
repo
github
repo
github
repo
user-api
timeline-api
relationship-api
search-api

@theburningmonk theburningmonk.com
CI/CD pipeline per service

@theburningmonk theburningmonk.com
functions are deployed together, as a stack

unencrypted secrets
in env vars
#7

@theburningmonk theburningmonk.com
secrets should NEVER be in plain text in env variables

@theburningmonk theburningmonk.com
SSM Parameter Store
Secret 1
Secret 2
IAM
Environment:
SECRET_1: …
SECRET_2: …
Environment:
SECRET_1: …
SECRET_2: …

@theburningmonk theburningmonk.com
SSM Parameter Store
Secret 1
Secret 2
IAM
Environment:
SECRET_1: …
SECRET_2: …
Environment:
SECRET_1: …
SECRET_2: …
yay!

@theburningmonk theburningmonk.com

@theburningmonk theburningmonk.com

@theburningmonk theburningmonk.com
SSM Parameter Store
Secret 1
Secret 2
IAM
fetch at cold start,
cache,
invalidate every x mins

@theburningmonk theburningmonk.com
https://github.com/middyjs/middy

@theburningmonk theburningmonk.com

@theburningmonk theburningmonk.com
SSM Parameter Store
Secret 1
Secret 2
IAM
switch to Higher
Throughput if you need
more than 40 ops/s

not following least
privilege principle
#8

@theburningmonk theburningmonk.com

@theburningmonk theburningmonk.com

missing DLQs
#9

@theburningmonk theburningmonk.com
async sync
S3
SNS
SES
CloudFormation
CloudWatch Logs
CloudWatch Events
Scheduled Events
CodeCommit
AWS Config
http://amzn.to/2v7Kc3b
Cognito
Alexa
Lex
API Gateway
pulling
DynamoDB Stream
Kinesis Stream
SQS

@theburningmonk theburningmonk.com
async sync
S3
SNS
SES
CloudFormation
CloudWatch Logs
CloudWatch Events
Scheduled Events
CodeCommit
AWS Config
http://amzn.to/2vs2lIg
Cognito
Alexa
Lex
API Gateway
pulling
DynamoDB Stream
Kinesis Stream
SQS
Lambda handles retries
(twice, then DLQ)

@theburningmonk theburningmonk.com
configure DLQ for async functions so you don’t lose failed events

@theburningmonk theburningmonk.com

@theburningmonk theburningmonk.com

@theburningmonk theburningmonk.com

@theburningmonk theburningmonk.com

too much/too little
concurrency
#10

@theburningmonk theburningmonk.com
“Lambda generates too much load for the downstream system”

@theburningmonk theburningmonk.com
one invocation
per message
SNS
Lambda

@theburningmonk theburningmonk.com
Downstream
System
SNS
Lambda

@theburningmonk theburningmonk.com
ordering
replay events
Kinesis SQS SNS
by shard
none (standard)
global (FIFO)
none
up to 7 days none none
mode
retry
batched batched (up to 10) singular
retried until
success
(customizable)
retry + DLQ retry + DLQ
concurrency 1 per shard auto-scaled fan-out!!!
subscribers many one-to-one many
EventBridge
many
none
none
singular
retry + DLQ
fan-out!!!

@theburningmonk theburningmonk.com
if you want…
maximum
throughput
SNS
precise control
over throughput
Kinesis

@theburningmonk theburningmonk.com
if you want…
maximum
throughput
SNS
precise control
over throughput
Kinesis
how quickly it scales out

@theburningmonk theburningmonk.com
if you want…
maximum
throughput
SNS
precise control
over throughput
Kinesis
how quickly it scales out
SQS DynamoDB
Streams

@theburningmonk theburningmonk.com
ordering
replay events
Kinesis SQS SNS
by shard
none (standard)
global (FIFO)
none
up to 7 days none none
mode
retry
batched batched (up to 10) singular
retried until
success
(customizable)
retry + DLQ retry + DLQ
concurrency 1 per shard auto-scaled fan-out!!!
subscribers many one-to-one many
EventBridge
many
none
none
singular
retry + DLQ
fan-out!!!

cold starts
#11

@theburningmonk theburningmonk.com
“cold starts only happen to the first request”

@theburningmonk theburningmonk.com
function invocationconcurrent execution
i.e. a container

@theburningmonk theburningmonk.com
function invocationconcurrent execution
i.e. a container
class instance method call

@theburningmonk theburningmonk.com
Lambda scales the number of concurrent executions
based on traffic

@theburningmonk theburningmonk.com
existing “containers” are reused where possible

@theburningmonk theburningmonk.com
time
invocation

@theburningmonk theburningmonk.com
time
invocation
invocation

@theburningmonk theburningmonk.com
time
invocation
invocation

@theburningmonk theburningmonk.com
time
invocation
invocation
invocation
invocation

@theburningmonk theburningmonk.com
time
invocation
invocation
invocation
invocation
invocation
invocation

@theburningmonk theburningmonk.com
time
invocation
invocation
invocation
invocation
invocation
invocation

@theburningmonk theburningmonk.com
time
invocation
invocation
invocation
invocation
invocation
invocation
invocation

@theburningmonk theburningmonk.com
time
invocation
invocation
invocation
invocation
invocation
invocation
invocation invocation

@theburningmonk theburningmonk.com
time
invocation
invocation
invocation
invocation
invocation
invocation
invocation invocation

@theburningmonk theburningmonk.com
time
invocation
invocation
invocation
invocation
invocation
invocation
invocation invocation

@theburningmonk theburningmonk.com

@theburningmonk theburningmonk.com

@theburningmonk theburningmonk.com

@theburningmonk theburningmonk.com

@theburningmonk theburningmonk.com
time
invocation
invocation
ping
invocation
invocation
invocation
ping ping

@theburningmonk theburningmonk.com
Lambda warmers don’t work when you have > 1
concurrent executions

@theburningmonk theburningmonk.com
FREQUENCY DURATION

@theburningmonk theburningmonk.com
FREQUENCY DURATION
dictated by user traffic,
out of your control

@theburningmonk theburningmonk.com
cold starts is generally not an issue if you have a
steady traffic pattern

@theburningmonk theburningmonk.com
time
req/s

@theburningmonk theburningmonk.com
time
req/s
El Classico

@theburningmonk theburningmonk.com
time
req/s
lunch dinner

@theburningmonk theburningmonk.com
FREQUENCY DURATION
optimize this!

@theburningmonk theburningmonk.com
minimise the duration of cold starts so they
fall within acceptable latency range

@theburningmonk theburningmonk.com
time
req/s
lunch dinner
Provisioned
Concurrency

@theburningmonk theburningmonk.com
time
req/s
lunch dinner
Provisioned
Concurrency
On-Demand
Concurrency

@theburningmonk theburningmonk.com
https://lumigo.io/blog/provisioned-concurrency-the-end-of-cold-starts/

@theburningmonk theburningmonk.com
there are no silver bullets

@theburningmonk theburningmonk.com
Provisioned concurrency is a powerful tool IFF you
have a cold start problem
don’t use it by default

RDS connection
handling
#12

@theburningmonk theburningmonk.com
default RDS configs are bad for Lambda

@theburningmonk theburningmonk.com
default RDS configs are bad for Lambda
idle connections are
not closed
too many connections
per “container”
max open connection
is too low

@theburningmonk theburningmonk.com
https://www.jeremydaly.com/manage-rds-connections-aws-lambda/

@theburningmonk theburningmonk.com
set “wait_timeout” and “interactive_timeout” to 10 mins
(default is 8 hours!)

@theburningmonk theburningmonk.com
increase “max_connections” setting

@theburningmonk theburningmonk.com
set client socket pool size to 1

@theburningmonk theburningmonk.com

@theburningmonk theburningmonk.com

(lack of) observability
#13

@theburningmonk theburningmonk.com
happened system repaireduser impact
reduce MTTR

@theburningmonk theburningmonk.com
Identify & Resolve
Issues
Understanding
costs
Visibility

@theburningmonk theburningmonk.com
Identify & Resolve
Issues
Understanding
costs
Visibility

@theburningmonk theburningmonk.com
happened system repaireduser impact
MTTDiscovery

@theburningmonk theburningmonk.com

@theburningmonk theburningmonk.com
“What alerts should I have?”

@theburningmonk theburningmonk.com
It depends on what you’re building…

@theburningmonk theburningmonk.com
But, this is a good starting point

@theburningmonk theburningmonk.com
Lambda
error rate %
throttle count
DLR error count
iterator age
regional concurrency

@theburningmonk theburningmonk.com
Lambda
error rate %
throttle count
DLR error count
iterator age
regional concurrency
API Gateway
p90/95/99 latency
success rate %
4xx rate %
5xx rate %

@theburningmonk theburningmonk.com
API Gateway
p90/95/99 latency
success rate %
4xx rate %
5xx rate %
SQS
message age
Lambda
error rate %
throttle count
DLR error count
iterator age
regional concurrency

@theburningmonk theburningmonk.com
API Gateway
p90/95/99 latency
success rate %
4xx rate %
5xx rate %
SQS
message age
Step Functions
failed count
throttle count
timed out count
Lambda
error rate %
throttle count
DLR error count
iterator age
regional concurrency

@theburningmonk theburningmonk.com
SQS
message age
Step Functions
failed count
throttle count
timed out count
API Gateway
p90/95/99 latency
success rate %
4xx rate %
5xx rate %
Lambda
error rate %
throttle count
DLR error count
iterator age
regional concurrency

@theburningmonk theburningmonk.com
monitor and alert on message flow rate for
event processing pipelines

@theburningmonk theburningmonk.com
“Can’t you codify these?”

@theburningmonk theburningmonk.com

@theburningmonk theburningmonk.com
not letting go of legacy thinking
one account that rules them all
do first, research later
not using a deployment framework
console-driven development
one repo per function
unencrypted secrets in env vars
not following least privilege principle
missing DLQs
too much/too little concurrency
cold starts
RDS connection handling
(lack of) observability
#1
#2
#3
#4
#5
#6
#7
#8
#9
#10
#11
#12
#13

https://theburningmonk.com/hire-me
AdviseTraining Delivery
“Fundamentally, Yan has improved our team by increasing our
ability to derive value from AWS and Lambda in particular.”
Nick Blair
Tech Lead

@theburningmonk theburningmonk.com
Production-Ready Serverless

in your
company
flexible datesHelsinki, Aug 20-21 London, Sep 24-25 Berlin, Oct 8-9Amsterdam, Jul 7-8
4-week virtual workshop, May 4 - May 29
@theburningmonk theburningmonk.com
theburningmonk.com/workshops
slsdays-virtual-202004
€100 off all my workshops

@theburningmonk theburningmonk.com
lambdabestpractice.com bit.ly/complete-guide-to-aws-step-functions
20% off my courses
slsdays-virtual-202004

@theburningmonk
theburningmonk.com
github.com/theburningmonk