How to build observability into a serverless application

how to build
Serverless
OBSERVABILITY
into a application

@theburningmonk theburningmonk.com
What do I mean by “observability”?

Monitoring
watching out for
known failure modes
in the system,
e.g. network I/O, CPU,
memory usage, …

Observability
being able to debug
the system, and gain
insights into the
system’s behaviour

In control theory, observability is a measure of how well
internal states of a system can be inferred from
knowledge of its external outputs.
https://en.wikipedia.org/wiki/Observability

In control theory, observability is a measure of how well
internal states of a system can be inferred from
knowledge of its external outputs.
https://en.wikipedia.org/wiki/Observability
including non-
functional outputs

These are the four pillars of the Observability Engineering
team’s charter:
• Monitoring
• Alerting/Visualization
• Distributed systems tracing infrastructure
• Log aggregation/analytics
“
” http://bit.ly/2DnjyuW- Observability Engineering at Twitter

microservices death stars circa 2015

mm… I wonder what’s
going on here…

I got this!

Yan Cui
http://theburningmonk.com
@theburningmonk
AWS user for 10 years

Yan Cui
@theburningmonk
Developer Advocate @

Yan Cui
@theburningmonk
Independent Consultant
advisetraining delivery

https://theburningmonk.com/courses
lambdabestpractice.com

theburningmonk.com/workshops
in your
company
ﬂexible datesHelsinki, Aug 20-21 London, Sep 24-25 Berlin, Oct 8-9
4-week virtual workshop, May 4 - May 29
Amsterdam, Jul 7-8

NO ACCESS
to underlying OS

NOWHERE
to install agents/daemons

•nowhere to install agents/daemons
new challenges

user request
user request
user request
user request
user request
user request
user request
critical paths:
minimise user-facing latency
handler
handler
handler
handler
handler
handler
handler

user request
user request
user request
user request
user request
user request
user request
critical paths:
StatsD
handler
handler
handler
handler
handler
handler
handler
rsyslog
background processing:
batched, asynchronous, low
overhead

user request
user request
user request
user request
user request
user request
user request
critical paths:
StatsD
handler
handler
handler
handler
handler
handler
handler
rsyslog
background processing:
batched, asynchronous, low
overhead
NO background processing
except what platform provides

•no background processing
new challenges

EC2
concurrency used to be
handled by your code

EC2
Lambda
Lambda
Lambda
Lambda
Lambda
now, it’s handled by the
AWS Lambda platform

EC2
logs & metrics used to be
batched here

EC2
Lambda
Lambda
Lambda
Lambda
Lambda
now, they are batched in each
concurrent execution, at best…

HIGHER concurrency to log
aggregation/telemetry system

•higher concurrency to telemetry system
new challenges

Lambda
cold start

Lambda
data is batched between
invocations

Lambda
idle
data is batched between
invocations

Lambda
idle
garbage collectiondata is batched between
invocations

Lambda
idle
garbage collectiondata is batched between
invocations
HIGH chance of data loss

•high chance of data loss (if batching)
new challenges

Lambda

my code
send metrics

my code
send metrics
internet internet
press button something happens

http://bit.ly/2Dpidje

?
functions are often chained together
via asynchronous invocations

?
SNS
Kinesis
CloudWatch
Events
CloudWatch
LogsIoT
DynamoDB
S3 SES

?
SNS
Kinesis
CloudWatch
Events
CloudWatch
LogsIoT
DynamoDB
S3 SES
tracing ASYNCHRONOUS
invocations through so many
different event sources is difficult

•asynchronous invocations
•high chance of data loss (if batching)
new challenges

2016-07-12T12:24:37.571Z 994f18f9-482b-11e6-8668-53e4eab441ae
GOT is off air, what do I do now?

2016-07-12T12:24:37.571Z 994f18f9-482b-11e6-8668-53e4eab441ae
GOT is off air, what do I do now?
UTC Timestamp Request Id
your log message

one log group per
function
one log stream for each
concurrent invocation

logs are not easily searchable in
CloudWatch Logs
me

CloudWatch Logs

CloudWatch Logs is an async event source for Lambda

Concurrent Executions
Time
regional max
concurrency
functions that are
delivering business value

Concurrent Executions
Time
regional max
concurrency
functions that are
delivering business value
ship logs

either set concurrency limit on the log shipping function
(and potentially lose logs due to throttling)
or…

1 shard = 1 concurrent execution
i.e. control the no. of concurrent
executions with no. of shards

…

https://amzn.to/2DnREgn

https://amzn.to/2uZYmEw

use structured logging with JSON

https://stackify.com/what-is-structured-logging-and-why-developers-need-it/ https://blog.treasuredata.com/blog/2012/04/26/log-everything-as-json/

https://www.loggly.com/blog/8-handy-tips-consider-logging-json/

traditional loggers are too heavy for Lambda

https://github.com/getndazn/dazn-lambda-powertools

Writing lots more data to
CloudWatch Logs

CloudWatch Logs
$0.50 per GB ingested
$0.03 per GB archived per month

CloudWatch Logs
$0.50 per GB ingested
$0.03 per GB archived per month
1M invocation of a 128MB function =
$0.000000208 * 1M + $0.20 =
$0.408

DON’T leave debug logging ON in production

have to redeploy ALL the
functions along the call path to
collect all relevant debug logs

https://github.com/middyjs/middy

EC2
Lambda
Lambda
Lambda
Lambda
Lambda
Concurrency is handled by
the AWS Lambda platform

sampling decision has to be
followed by an entire call chain

Initial Request ID
User ID
Session ID
User-Agent
Order ID
…

store correlation IDs in global variable

use middleware to auto-capture incoming correlation IDs

extract correlation IDs from
invocation event, and store them in
the correlation-ids module
reset

logger to always include captured correlation IDs

HTTP and AWS SDK clients to auto-forward correlation IDs on

context.awsRequestId
get-index

context.awsRequestId x-correlation-id
get-index

{
“headers”: {
“x-correlation-id”: “…”
},
…
}
get-index

{
“body”: null,
“resource”: “/restaurants”,
“headers”: {
“x-correlation-id”: “…”
},
…
}
get-index get-restaurants

get-restaurants
global.CONTEXT
global.CONTEXT
x-correlation-id = …
x-correlation-xxx = …
get-index
headers[“User-Agent”]
headers[“Debug-Log-Enabled”]
headers[“User-Agent”]
headers[“Debug-Log-Enabled”]
headers[“x-correlation-id”]
capture
forward
function
event
log.info(…)

MONITORING

those extra 10-20ms for
sending custom metrics would
compound when you have
microservices and multiple
APIs are called within one slice
of user event

Amazon found every 100ms of latency cost them 1% in sales.
http://bit.ly/2EXPfbA

https://aws.amazon.com/cloudwatch/pricing/

https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/cloudwatch_concepts.html

happened system repaireduser impact
reduce MTTR

Identify & Resolve
Issues
Understanding
costs
Visibility

happened system repaireduser impact
MTTDiscovery

“What alerts should I have?”

It depends on what you’re building…

But, this is a good starting point

Lambda
error rate %
throttle count
DLR error count
iterator age
regional concurrency

Lambda
error rate %
throttle count
DLR error count
iterator age
API Gateway
p90/95/99 latency
success rate %
4xx rate %
5xx rate %

API Gateway
p90/95/99 latency
success rate %
4xx rate %
5xx rate %
SQS
message age
Lambda
error rate %
throttle count
DLR error count
iterator age

API Gateway
p90/95/99 latency
success rate %
4xx rate %
5xx rate %
SQS
message age
Step Functions
failed count
throttle count
timed out count
Lambda
error rate %
throttle count
DLR error count
iterator age

SQS
message age
Step Functions
failed count
throttle count
timed out count
API Gateway
p90/95/99 latency
success rate %
4xx rate %
5xx rate %
Lambda
error rate %
throttle count
DLR error count
iterator age

monitor and alert on message ﬂow rate for
event processing pipelines

“Can’t you codify these?”

TRACING

poor support for async
invocations
good for identifying dependencies of a function,
but not good enough for tracing the entire call
chain as user request/data flows through the
system via async event sources.

don’t span over non-AWS services

make it easy to do the right thing

https://theburningmonk.com/hire-me
AdviseTraining Delivery
“Fundamentally, Yan has improved our team by increasing our
ability to derive value from AWS and Lambda in particular.”
Nick Blair
Tech Lead

Production-Ready Serverless

theburningmonk.com/workshops
in your
company
ﬂexible datesHelsinki, Aug 20-21 London, Sep 24-25 Berlin, Oct 8-9
4-week virtual workshop, May 4 - May 29
Amsterdam, Jul 7-8
smartly-2020
€100 off all my workshops

lambdabestpractice.com bit.ly/complete-guide-to-aws-step-functions
smartly-2020
20% off my courses

@theburningmonk
theburningmonk.com
github.com/theburningmonk

How to build observability into a serverless application

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to How to build observability into a serverless application

Similar to How to build observability into a serverless application (20)

More from Yan Cui

More from Yan Cui (20)

Recently uploaded

Recently uploaded (20)

How to build observability into a serverless application