AWS Lambda from the trenches (Serverless London)

Yan Cui
Yan CuiSpeaker at Self
from the
TRENCHESTRENCHES
what you should know before you go to production
AWS LAMBDAAWS LAMBDA
hi,I’mYanCui
hi,I’mYanCui
AWS user since 2009
AWS Lambda from the trenches (Serverless London)
AWS Lambda from the trenches (Serverless London)
AWS Lambda from the trenches (Serverless London)
AWS Lambda from the trenches (Serverless London)
AWS Lambda from the trenches (Serverless London)
AWS Lambda from the trenches (Serverless London)
apr, 2016
hidden complexities and dependencies
low utilisation to leave room for traffic spikes
EC2 scaling is slow, so scale earlier
lots of cost for unused resources
up to 30 mins for deployment
deployment required downtime
- Dan North
“lead time to someone saying
thank you is the only reputation
metric that matters.”
AWS Lambda from the trenches (Serverless London)
“what would good
look like for us?”
be small
be fast
have zero downtime
have no lock-step
DEPLOYMENTS SHOULD...
FEATURES SHOULD...
be deployable independently
be loosely-coupled
WE WANT TO...
minimise cost for unused resources
minimise ops effort
reduce tech mess
deliver visible improvements faster
nov, 2016
170 Lambda functions in prod
1.2 GB deployment packages in prod
95% cost saving vs EC2
15x no. of prod releases per month
time
is a good fit
1st function in prod!
time
is a good fit
?
time
is a good fit
1st function in prod!
Practices ToolsPrinciples
what is good? how to make it good? with what?
Principles outlast Tools
ALERTING
CI / CD
TESTING
LOGGING
MONITORING
170 functions
WOOF!
? ?
time
is a good fit
1st function in prod!
SECURITY
DISTRIBUTED
TRACING
CONFIG
MANAGEMENT
evolving the PLATFORM
rebuilt search
Legacy Monolith Amazon Kinesis Amazon Lambda
Amazon CloudSearch
Legacy Monolith Amazon Kinesis Amazon Lambda
Amazon CloudSearchAmazon API Gateway Amazon Lambda
new analytics pipeline
Legacy Monolith Amazon Kinesis Amazon Lambda
Google BigQuery
Legacy Monolith Amazon Kinesis Amazon Lambda
Google BigQuery
1 developer, 2 days
design production
(his 1st serverless project)
Legacy Monolith Amazon Kinesis Amazon Lambda
Google BigQuery
“nothing ever got done
this fast at Skype!”
- Chris Twamley
- Dan North
“lead time to someone saying
thank you is the only reputation
metric that matters.”
Rebuilt
with Lambda
AWS Lambda from the trenches (Serverless London)
AWS Lambda from the trenches (Serverless London)
AWS Lambda from the trenches (Serverless London)
AWS Lambda from the trenches (Serverless London)
AWS Lambda from the trenches (Serverless London)
AWS Lambda from the trenches (Serverless London)
Rebuilt
with Lambda
BigQuery
BigQuery
grapheneDB
BigQuery
grapheneDB
BigQuery
grapheneDB
BigQuery
getting PRODUCTION READY
CHOOSE A
FRAMEWORK
DEPLOYMENT
http://serverless.com
https://github.com/awslabs/serverless-application-model
http://apex.run
https://apex.github.io/up
https://github.com/claudiajs/claudia
https://github.com/Miserlou/Zappa
http://gosparta.io/
TESTING
amzn.to/29Lxuzu
Level of Testing
1.Unit
do our objects do the right thing?
are they easy to work with?
AWS Lambda from the trenches (Serverless London)
Level of Testing
1.Unit
2.Integration
does our code work against code we
can’t change?
handler
handler
test by invoking
the handler
Level of Testing
1.Unit
2.Integration
3.Acceptance
does the whole system work?
Level of Testing
unit
integration
acceptance
feedback
confidence
“…We find that tests that mock external
libraries often need to be complex to
get the code into the right state for the
functionality we need to exercise.
The mess in such tests is telling us that
the design isn’t right but, instead of
fixing the problem by improving the
code, we have to carry the extra
complexity in both code and test…”
Don’t Mock Types You Can’t Change
“…The second risk is that we have to be
sure that the behaviour we stub or mock
matches what the external library will
actually do…
Even if we get it right once, we have to
make sure that the tests remain valid
when we upgrade the libraries…”
Don’t Mock Types You Can’t Change
Don’t Mock Types You Can’t Change
Services
Paul Johnston
The serverless approach to
testing is different and may
actually be easier.
http://bit.ly/2t5viwK
LambdaAPI Gateway DynamoDB
LambdaAPI Gateway DynamoDB
Unit Tests
LambdaAPI Gateway DynamoDB
Unit Tests
Mock/Stub
is our request correct?
is the request mapping
set up correctly?is the API resources
configured correctly?
are we assuming the
correct schema?
LambdaAPI Gateway DynamoDB
is Lambda proxy
configured correctly?
is IAM policy set
up correctly?
is the table created?
what unit tests will not tell you…
AWS Lambda from the trenches (Serverless London)
most Lambda functions are simple
have single purpose, the risk of
shipping broken software has largely
shifted to how they integrate with
external services
observation
AWS Lambda from the trenches (Serverless London)
But it slows down
my feedback loop…
IT’S NOT
ABOUT YOU!
AWS Lambda from the trenches (Serverless London)
IT’S CHINA.
NOT SCHINA.
AWS Lambda from the trenches (Serverless London)
AWS Lambda from the trenches (Serverless London)
AWS Lambda from the trenches (Serverless London)
AWS Lambda from the trenches (Serverless London)
AWS Lambda from the trenches (Serverless London)
AWS Lambda from the trenches (Serverless London)
me
Your users shouldn’t be the
ones to pay the price for your
faster feedback loop.
Optimise for working software.
Test your software end-to-end.
“…Wherever possible, an acceptance
test should exercise the system end-to-
end without directly calling its internal
code.
An end-to-end test interacts with the
system only from the outside: through
its interface…”
Testing End-to-End
Legacy Monolith Amazon Kinesis Amazon Lambda
Amazon CloudSearchAmazon API Gateway Amazon Lambda
Legacy Monolith Amazon Kinesis Amazon Lambda
Amazon CloudSearchAmazon API Gateway Amazon Lambda
Test Input
Legacy Monolith Amazon Kinesis Amazon Lambda
Amazon CloudSearchAmazon API Gateway Amazon Lambda
Test Input
Validate
integration tests exercise
system’s Integration with its
external dependencies
acceptance tests exercise
system End-to-End from
the outside
integration tests differ from
acceptance tests only in HOW the
Lambda functions are invoked
observation
AWS Lambda from the trenches (Serverless London)
AWS Lambda from the trenches (Serverless London)
AWS Lambda from the trenches (Serverless London)
CI + CD PIPELINE
“the earlier you consider CI + CD, the
more time you save in the long run”
- me
“…We prefer to have the end-to-end
tests exercise both the system and the
process by which it’s built and
deployed…
This sounds like a lot of effort (it is), but
has to be done anyway repeatedly
during the software’s lifetime…”
Testing End-to-End
“deployment scripts
that only live on the CI
box is a disaster
waiting to happen”
- me
Jenkins build config deploys and tests
unit + integration tests
deploy
acceptance tests
if [ "$1" = "deploy" ] && [ $# -eq 4 ]; then
STAGE=$2
REGION=$3
PROFILE=$4
npm install
AWS_PROFILE=$PROFILE 'node_modules/.bin/sls' deploy -s $STAGE -r $REGION
elif [ "$1" = "int-test" ] && [ $# -eq 4 ]; then
STAGE=$2
REGION=$3
PROFILE=$4
npm install
AWS_PROFILE=$PROFILE npm run int-$STAGE
elif [ "$1" = "acceptance-test" ] && [ $# -eq 4 ]; then
STAGE=$2
REGION=$3
PROFILE=$4
npm install
AWS_PROFILE=$PROFILE npm run acceptance-$STAGE
else
usage
exit 1
fi
build.sh allows repeatable builds on both local & CI
AWS Lambda from the trenches (Serverless London)
Auto Auto Manual
LOGGING
AWS Lambda from the trenches (Serverless London)
2016-07-12T12:24:37.571Z 994f18f9-482b-11e6-8668-53e4eab441ae
GOT is off air, what do I do now?
2016-07-12T12:24:37.571Z 994f18f9-482b-11e6-8668-53e4eab441ae
GOT is off air, what do I do now?
UTC Timestamp API Gateway Request Id
your log message
function name
date
function version
LOG OVERLOAD
CENTRALISE LOGS
CENTRALISE LOGS
MAKE THEM EASILY
SEARCHABLE
+ +
the elk stack
CloudWatch Logs
CloudWatch Logs AWS Lambda ELK stack
CloudWatch Events
AWS Lambda from the trenches (Serverless London)
http://bit.ly/2f3zxQG
DISTRIBUTED TRACING
AWS Lambda from the trenches (Serverless London)
“my followers didn’t
receive my new post!”
- a user
where could the
problem be?
correlation IDs*
* eg. request-id, user-id, yubl-id, etc.
ROLL YOUR OWN
CLIENTS
kinesis client
http client
sns client
http://bit.ly/2k93hAj
ROLL YOUR OWN
CLIENTS
X-RAY
Amazon X-Ray
Amazon X-Ray
traces do not span over
API Gateway
http://bit.ly/2s9yxmA
MONITORING + ALERTING
“where do I install
monitoring agents?”
you can’t
• invocation Count
• error Count
• latency
• throttling
• granular to the minute
• support custom metrics
• same metrics as CW
• better dashboard
• support custom metrics
https://www.datadoghq.com/blog/monitoring-lambda-functions-datadog/
AWS Lambda from the trenches (Serverless London)
“how do I batch up
and send logs in the
background?”
you can’t
(kinda)
console.log(“hydrating yubls from db…”);
console.log(“fetching user info from user-api”);
console.log(“MONITORING|1489795335|27.4|latency|user-api-latency”);
console.log(“MONITORING|1489795335|8|count|yubls-served”);
timestamp metric value
metric type
metric namemetrics
logs
CloudWatch Logs AWS Lambda
ELK stack
logs
metrics
CloudWatch
http://bit.ly/2gGredx
DASHBOARDS
DASHBOARDS
SET ALARMS
DASHBOARDS
SET ALARMS
TRACK APP-LEVEL
METRICS
Not Only CloudWatch
AWS Lambda from the trenches (Serverless London)
“you really don't want
your monitoring
system to fail at the
same time as the
system it monitors”
- me
CONFIG MANAGEMENT
easily and quickly propagate
config changes
AWS Lambda from the trenches (Serverless London)
CENTRALISED
CONFIG SERVICE
config service
goes here
AWS Lambda from the trenches (Serverless London)
AWS Lambda from the trenches (Serverless London)
AWS Lambda from the trenches (Serverless London)
SSM
Parameter
Store
sensitive data should be encrypted
in-flight, and at rest
(credentials, connection string, etc.)
role-based access
SSM Parameter Store
HTTPS
role-based access
encrypted in-flight
SSM Parameter Store
encrypt
role-based access
SSM Parameter Store
encrypted at-rest
HTTPS
role-based access
SSM Parameter Store
encrypted in-flight
CENTRALISED
CONFIG SERVICE
CLIENT LIBRARY
fetch & cache at Cold Start
invalidate at interval + signal
http://bit.ly/2yLUjwd
PRO TIPS
max 75 GB total deployment package size*
* limit is per AWS region
Janitor Monkey
Janitor Lambda
http://bit.ly/2xzVu4a
disable versionFunctions in
install Serverless framework as dev
dependency at project level
dev dependencies are excluded since 1.16.0
http://bit.ly/2vzBqhC
http://amzn.to/2vtUkDU
UNDERSTAND
COLDSTARTS
Amazon X-Ray
1st invocation
2nd invocation
cold start
source: http://bit.ly/2oBEbw2
EMBRACE
NODE.JS & PYTHON
http://bit.ly/2rtCCBz
C#
http://bit.ly/2rtCCBz
Java
http://bit.ly/2rtCCBz
NodeJs, Python
http://bit.ly/2rtCCBz
what about type safety?
AWS Lambda from the trenches (Serverless London)
complexity ceiling of a
Node.js app
complexity
complexity ceiling of a
Node.js app
complexity
referential transparency
immutability as default
type inference
option types
union types
…
for managing complexity
complexity ceiling of a
Node.js app
complexity
referential transparency
immutability as default
type inference
option types
union types
…
complexity ceiling of a
Node.js app
complexity
complexity ceiling of a
Node.js Lambda function
if you can limit the complexity
of your solution, maybe you
won’t need the tools for
managing that complexity.
me
AVOID HARD
ASSUMPTIONS
ABOUT FUNCTION
LIFETIME
USE STATE
FOR
OPTIMISATION
AVOID
COLDSTARTS
CloudWatch Event AWS Lambda
CloudWatch Event AWS Lambda
ping
ping
ping
ping
CloudWatch Event AWS Lambda
ping
ping
ping
ping
CloudWatch Event AWS Lambda
ping
ping
ping
ping
HEALTH CHECKS?
max 5 mins execution time
USE RECURSION
FOR LONG
RUNNING TASKS
CONSIDER
PARTIAL
FAILURES
“AWS Lambda polls your stream and
invokes your Lambda function. Therefore, if
a Lambda function fails, AWS Lambda
attempts to process the erring batch of
records until the time the data expires…”
http://docs.aws.amazon.com/lambda/latest/dg/retries-on-errors.html
should function fail on
partial/any failures?
SNS
Kinesis
SQS
after 3 attempts
share processing logic
events are processed in
chronological order
failed events are retried out
of sequence
PROCESS SQS
WITH RECURSIVE
FUNCTIONS
http://bit.ly/2npomX6
AVOID HOT
KINESS
STREAMS
“Each shard can support up to
5 transactions per second for
reads, up to a maximum total data
read rate of 2 MB per second.”
http://docs.aws.amazon.com/streams/latest/dev/service-sizes-and-limits.html
“If your stream has 100 active shards,
there will be 100 Lambda functions
running concurrently. Then, each
Lambda function processes events
on a shard in the order that they arrive.”
http://docs.aws.amazon.com/lambda/latest/dg/concurrent-executions.html
when no. of processors goes up…
ReadProvisionedThroughputExceeded
can have too many Kinesis read operations…
ReadRecords.IteratorAge
unpredictable spikes in read ‘latency’…
can kinda workaround…
http://bit.ly/2uv5LsH
clever, but costly
for subsystems that don’t have
to be realtime, or are task-
based (ie. order doesn’t
matter), consider other
triggers such as S3 or SNS.me
@theburningmonk
theburningmonk.com
github.com/theburningmonk
@theburningmonk
theburningmonk.com
github.com/theburningmonk
http://bit.ly/2yQZj1H
all my blog posts on Lambda
sign up here: http://bit.ly/2xIO23O
1 of 221

More Related Content

What's hot(20)

A Walk in the Cloud with AWS LambdaA Walk in the Cloud with AWS Lambda
A Walk in the Cloud with AWS Lambda
Amazon Web Services10.4K views
AWS Lambda: Event-driven Code in the CloudAWS Lambda: Event-driven Code in the Cloud
AWS Lambda: Event-driven Code in the Cloud
Amazon Web Services5K views
Amazon ECS Deep DiveAmazon ECS Deep Dive
Amazon ECS Deep Dive
Amazon Web Services1.6K views
Getting Started with Docker on AWSGetting Started with Docker on AWS
Getting Started with Docker on AWS
Amazon Web Services596 views
AWS LambdaAWS Lambda
AWS Lambda
Danilo Poccia2.5K views
5 Things You Don't Know About AWS Cloud5 Things You Don't Know About AWS Cloud
5 Things You Don't Know About AWS Cloud
Amazon Web Services4.4K views
Serverless for DevelopersServerless for Developers
Serverless for Developers
Amazon Web Services1.1K views
Getting Started with Serverless ArchitecturesGetting Started with Serverless Architectures
Getting Started with Serverless Architectures
Amazon Web Services9.3K views
A Walk in the Cloud with AWS LambdaA Walk in the Cloud with AWS Lambda
A Walk in the Cloud with AWS Lambda
Amazon Web Services10.3K views
Deep Dive on AWS LambdaDeep Dive on AWS Lambda
Deep Dive on AWS Lambda
Amazon Web Services6.3K views
AWS Lambda and Serverless CloudAWS Lambda and Serverless Cloud
AWS Lambda and Serverless Cloud
Amazon Web Services7.6K views

Similar to AWS Lambda from the trenches (Serverless London)(20)

AWS Lambda from the trenchesAWS Lambda from the trenches
AWS Lambda from the trenches
Yan Cui5.3K views

Recently uploaded(20)

METHOD AND SYSTEM FOR PREDICTING OPTIMAL LOAD FOR WHICH THE YIELD IS MAXIMUM ...METHOD AND SYSTEM FOR PREDICTING OPTIMAL LOAD FOR WHICH THE YIELD IS MAXIMUM ...
METHOD AND SYSTEM FOR PREDICTING OPTIMAL LOAD FOR WHICH THE YIELD IS MAXIMUM ...
Prity Khastgir IPR Strategic India Patent Attorney Amplify Innovation24 views
Web Dev - 1 PPT.pdfWeb Dev - 1 PPT.pdf
Web Dev - 1 PPT.pdf
gdsczhcet49 views
Liqid: Composable CXL PreviewLiqid: Composable CXL Preview
Liqid: Composable CXL Preview
CXL Forum120 views
CXL at OCPCXL at OCP
CXL at OCP
CXL Forum203 views
Java Platform Approach 1.0 - Picnic MeetupJava Platform Approach 1.0 - Picnic Meetup
Java Platform Approach 1.0 - Picnic Meetup
Rick Ossendrijver24 views

AWS Lambda from the trenches (Serverless London)