The document discusses the challenges and advancements in serverless observability, particularly in the context of AWS Lambda. It emphasizes the importance of understanding failure modes while avoiding survivor bias in monitoring systems, and outlines the four pillars of observability engineering: monitoring, alerting/visualization, distributed systems tracing, and log aggregation. The necessity for improved tools and techniques to trace asynchronous invocations and to monitor system health at a granular event level is also highlighted.

1. @theburningmonk#aws #awslambda #serverless
the present and future of
serverless observability
Yan Cui @theburningmonk

3. Abraham Wald

4. Abraham Wald

5. Abraham Wald

6. Abraham Wald
Wald noted that the study only
considered the aircraft that had survived
their missions—the bombers that had
been shot down were not present for the
damage assessment.
The holes in the returning aircraft, then,
represented areas where a bomber could
take damage and still return home safely.

7. Abraham Wald
Wald noted that the study only
considered the aircraft that had survived
their missions—the bombers that had
been shot down were not present for the
damage assessment.
The holes in the returning aircraft, then,
represented areas where a bomber could
take damage and still return home safely.

8. survivor bias in monitoring

9. survivor bias in monitoring
Only focus on failure modes that we were able to successfully
identify through investigation and postmortem in the past.
The bullet holes that shot us down and we couldn’t identify stay
invisible, and will continue to shoot us down.

10. Yan Cui
http://theburningmonk.com
@theburningmonk
Principal Engineer @

15. available in Austria, Switzerland, Germany,
Japan, Canada, Italy and US

16. available on 30+ platforms

17. ~1,000,000 concurrent viewers

18. We’re hiring! Visit
engineering.dazn.com
to learn more.
follow @dazneng for
updates about the
engineering team

19. follow @dazneng for
updates about the
engineering team
We’re hiring! Visit
engineering.dazn.com
to learn more.
WE’RE HIRING!

20. AWS user since 2009

21. AWS user since 2009

22. https://www.youtube.com/watch?v=pptsgV4bKv8

23. https://bit.ly/production-ready-serverless

25. http://bit.ly/2C9LwIM

26. 2017
observability

27. Monitoring
watching out for
known failure modes
in the system,
e.g. network I/O, CPU,
memory usage, …

28. Observability
being able to debug
the system, and gain
insights into the
system’s behaviour

29. In control theory, observability is a measure of how well
internal states of a system can be inferred from
knowledge of its external outputs.
https://en.wikipedia.org/wiki/Observability

30. Known Success

31. Known SuccessKnown Errors

32. Known SuccessKnown Errors
easy to monitor!

33. Known SuccessKnown Errors
Known Unknowns

34. Known SuccessKnown Errors
Known UnknownsUnknown Unknowns

35. Known SuccessKnown Errors
Known UnknownsUnknown Unknowns
invisible bullet
holes

36. Known SuccessKnown Errors
Known UnknownsUnknown Unknowns

37. Known SuccessKnown Errors
Known UnknownsUnknown Unknowns
only alert on
this

38. Known SuccessKnown Errors
Known UnknownsUnknown Unknowns
alert on the
absence of this!

39. Known SuccessKnown Errors
Known UnknownsUnknown Unknowns
what went wrong?

40. These are the four pillars of the Observability Engineering
team’s charter:
• Monitoring
• Alerting/visualization
• Distributed systems tracing infrastructure
• Log aggregation/analytics
“
” http://bit.ly/2DnjyuW- Observability Engineering at Twitter

41. microservices death stars circa 2015

46. microservices death stars circa 2015
I got this!

48. new
challenges

49. new
challenges

50. NO ACCESS
to underlying OS

51. NOWHERE
to install agents/daemons

52. •nowhere to install agents/daemons
new challenges

53. user request
user request
user request
user request
user request
user request
user request
critical paths:
minimise user-facing latency
handler
handler
handler
handler
handler
handler
handler

54. user request
user request
user request
user request
user request
user request
user request
critical paths:
minimise user-facing latency
StatsD
handler
handler
handler
handler
handler
handler
handler
rsyslog
background processing:
batched, asynchronous, low
overhead

55. user request
user request
user request
user request
user request
user request
user request
critical paths:
minimise user-facing latency
StatsD
handler
handler
handler
handler
handler
handler
handler
rsyslog
background processing:
batched, asynchronous, low
overhead
NO background processing
except what platform provides

56. •no background processing
•nowhere to install agents/daemons
new challenges

57. EC2
concurrency used to be
handled by your code

58. EC2
Lambda
Lambda
Lambda
Lambda
Lambda
now, it’s handled by the
AWS Lambda platform

59. EC2
logs & metrics used to be
batched here

60. EC2
Lambda
Lambda
Lambda
Lambda
Lambda
now, they are batched in each
concurrent execution, at best…

63. HIGHER concurrency to log
aggregation/telemetry system

64. •higher concurrency to telemetry system
•nowhere to install agents/daemons
•no background processing
new challenges

65. Lambda
cold start

66. Lambda
data is batched between
invocations

67. Lambda
idle
data is batched between
invocations

68. Lambda
idle
garbage collectiondata is batched between
invocations

69. Lambda
idle
garbage collectiondata is batched between
invocations
HIGH chance of data loss

70. •high chance of data loss (if batching)
•nowhere to install agents/daemons
•no background processing
•higher concurrency to telemetry system
new challenges

71. Lambda

72. my code
send metrics

73. my code
send metrics

74. my code
send metrics
internet internet
press button something happens

76. http://bit.ly/2Dpidje

77. ?
functions are often chained together
via asynchronous invocations

78. ?
SNS
Kinesis
CloudWatch
Events
CloudWatch
LogsIoT
DynamoDB
S3 SES

79. ?
SNS
Kinesis
CloudWatch
Events
CloudWatch
LogsIoT
DynamoDB
S3 SES
tracing ASYNCHRONOUS
invocations through so many
different event sources is difficult

80. •asynchronous invocations
•nowhere to install agents/daemons
•no background processing
•higher concurrency to telemetry system
•high chance of data loss (if batching)
new challenges

81. the Present

82. These are the four pillars of the Observability Engineering
team’s charter:
• Monitoring
• Alerting/visualization
• Distributed systems tracing infrastructure
• Log aggregation/analytics
“
” http://bit.ly/2DnjyuW- Observability Engineering at Twitter

85. 2016-07-12T12:24:37.571Z 994f18f9-482b-11e6-8668-53e4eab441ae
GOT is off air, what do I do now?

86. 2016-07-12T12:24:37.571Z 994f18f9-482b-11e6-8668-53e4eab441ae
GOT is off air, what do I do now?
UTC Timestamp Request Id
your log message

88. one log group per
function
one log stream for each
concurrent invocation

89. logs are not easily searchable in
CloudWatch Logs
me

90. CloudWatch Logs

91. CloudWatch Logs AWS Lambda ELK stack

92. …

93. CloudWatch Logs

99. CloudWatch Logs

101. •no background processing
•nowhere to install agents/daemons
new challenges

102. my code
send metrics
internet internet
press button something happens

103. those extra 10-20ms for
sending custom metrics would
compound when you have
microservices and multiple
APIs are called within one slice
of user event

104. Amazon found every 100ms of latency cost them 1% in sales.
http://bit.ly/2EXPfbA

105. console.log(“hydrating yubls from db…”);
console.log(“fetching user info from user-api”);
console.log(“MONITORING|1489795335|27.4|latency|user-api-latency”);
console.log(“MONITORING|1489795335|8|count|yubls-served”);
timestamp metric value
metric type
metric namemetrics
logs

106. CloudWatch Logs AWS Lambda
ELK stack
logs
m
etrics
CloudWatch

108. delay
cost
concurrency

109. delay
cost
concurrency
no latency
overhead

110. API Gateway
send custom metrics
asynchronously

111. SNS KinesisS3API Gateway
…
send custom metrics
asynchronously
send custom metrics as
part of function invocation

113. X-Ray

116. do not span over API Gateway

117. narrow focus on a function
good for homing in on performance issues
for a particular function, but offers little to
help you build intuition about how your
system operates as a whole.

118. However, I would argue that the health of the system no
longer matters. We've entered an era where what matters is
the health of each individual event, or each individual user's
experience, or each shopping cart's experience (or other high
cardinality dimensions). With distributed systems you don't
care about the health of the system, you care about the
health of the event or the slice.
”http://bit.ly/2E2QngU- Charity Majors
“

119. follow the data

121. don’t span over async invocations
good for identifying dependencies of a function,
but not good enough for tracing the entire call
chain as user request/data flows through the
system via async event sources.

122. don’t span over non-AWS services

124. static view

125. our tools need to do more to help us with
understanding & debugging our distributed system,
not just what happens inside one function

127. “one user action/vertical slice through the system”

128. microservices death stars circa 2015

129. microservices death stars circa 2015
HELP…

131. WARNING: this is part fiction, part inspired by new tools

132. DASHBOARDS

136. different dimensions of service X
splattered across the screen

139. + cold starts
+ throttled invocations
+ concurrent executions
+ estimated cost ($)

140. SubscriberGetAccount
200,545
0
19
94
0
0 %
0 %
Est Cost:
Req Rate:
$54.0/s
20,056.0/s
Concurrency
Median
Mean 99.5th
99th
90th370
1ms
4ms 61ms
44ms
10ms
circle colour and size represent
health and traffic volume
2 minutes of request rate to
show relative changes in traffic
no. of concurrent executions
of this function
Request rate
Estimated cost
Error percentage
of last 10 seconds
Cold start percentage
last 10 seconds
last minute latency percentiles
200,545
0
19
94
0
Rolling 10 second counters
with 1 second granularity
Successes
Cold starts
Timeouts
Throttled Invocations
Errors

141. SubscriberGetAccount
200,545
0
19
94
0
0 %
0 %
Est Cost:
Req Rate:
$54.0/s
20,056.0/s
Concurrency
Median
Mean 99.5th
99th
90th370
1ms
4ms 61ms
44ms
10ms
circle colour and size represent
health and traffic volume
2 minutes of request rate to
show relative changes in traffic
no. of concurrent executions
of this function
Request rate
Estimated cost
Error percentage
of last 10 seconds
Cold start percentage
last 10 seconds
last minute latency percentiles
200,545
0
19
94
0
Rolling 10 second counters
with 1 second granularity
Successes
Cold starts
Timeouts
Throttled Invocations
Errors

142. SubscriberGetAccount
200,545
0
19
94
0
0 %
0 %
Est Cost:
Req Rate:
$54.0/s
20,056.0/s
Concurrency
Median
Mean 99.5th
99th
90th370
1ms
4ms 61ms
44ms
10ms
circle colour and size represent
health and traffic volume
2 minutes of request rate to
show relative changes in traffic
no. of concurrent executions
of this function
Request rate
Estimated cost
Error percentage
of last 10 seconds
Cold start percentage
last 10 seconds
last minute latency percentiles
200,545
0
19
94
0
Rolling 10 second counters
with 1 second granularity
Successes
Cold starts
Timeouts
Throttled Invocations
Errors

143. SubscriberGetAccount
200,545
0
19
94
0
0 %
0 %
Est Cost:
Req Rate:
$54.0/s
20,056.0/s
Concurrency
Median
Mean 99.5th
99th
90th370
1ms
4ms 61ms
44ms
10ms
circle colour and size represent
health and traffic volume
2 minutes of request rate to
show relative changes in traffic
no. of concurrent executions
of this function
Request rate
Estimated cost
Error percentage
of last 10 seconds
Cold start percentage
last 10 seconds
last minute latency percentiles
200,545
0
19
94
0
Rolling 10 second counters
with 1 second granularity
Successes
Cold starts
Timeouts
Throttled Invocations
Errors

144. SubscriberGetAccount
200,545
0
19
94
0
0 %
0 %
Est Cost:
Req Rate:
$54.0/s
20,056.0/s
Concurrency
Median
Mean 99.5th
99th
90th370
1ms
4ms 61ms
44ms
10ms
circle colour and size represent
health and traffic volume
2 minutes of request rate to
show relative changes in traffic
no. of concurrent executions
of this function
Request rate
Estimated cost
Error percentage
of last 10 seconds
Cold start percentage
last 10 seconds
last minute latency percentiles
200,545
0
19
94
0
Rolling 10 second counters
with 1 second granularity
Successes
Cold starts
Timeouts
Throttled Invocations
Errors

150. birds-eye view of our system as it lives and breathes

152. user
profile-images
POST /user
process-images
resize-images
image-tasks
Auth0
create-user
reformat-imagestag-user
Face API
create-auth0-user

153. user
profile-images
POST /user
process-images
resize-images
image-tasks
Auth0
create-user
reformat-imagestag-user
Face API
trace async invocations
create-auth0-user

154. user
profile-images
POST /user
process-images
resize-images
image-tasks
Auth0
create-user
reformat-imagestag-user
Face API
trace non-AWS resources
create-auth0-user

155. user
profile-images
POST /user
process-images
resize-images
image-tasks
Auth0
create-user
reformat-imagestag-user
Face API
Logs
timestamp component message
POST /user2018/01/25 20:51:23.188
2018/01/25 20:51:23.201 create-user
2018/01/25 20:51:23.215 create-user
2018/01/25 20:51:23.521 tag-user
incoming request…
saving user [theburningmonk] in the [user] table…
saved user [theburningmonk] in the [user] table
level
debug
debug
debug
debug tagging user [theburningmonk] with Azure Face API…
create-auth0-user

156. user
profile-images
POST /user
process-images
resize-images
image-tasks
Auth0
create-user
reformat-imagestag-user
Face API
Logs
timestamp component message
POST /user2018/01/25 20:51:23.188
2018/01/25 20:51:23.201 create-user
2018/01/25 20:51:23.215 create-user
2018/01/25 20:51:23.521 tag-user
incoming request…
saving user [theburningmonk] in the [user] table…
saved user [theburningmonk] in the [user] table
level
debug
debug
debug
debug tagging user [theburningmonk] with Azure Face API…
create-auth0-user

157. user
profile-images
POST /user
process-images
resize-images
image-tasks
Auth0
create-user
reformat-imagestag-user
Face API
Logs
timestamp component message
POST /user2018/01/25 20:51:23.188 incoming request…
level
debug
request-id
start-time
0ae4ba5d-dab1-4f9e-9de7-eace27ebfbc2
2018/01/25 20:51:23.188
method POST
create-auth0-user

158. user
profile-images
POST /user
process-images
resize-images
image-tasks
Auth0
create-user
reformat-imagestag-user
Face API
Logs
timestamp component message
2018/01/25 20:51:23.201 create-user
2018/01/25 20:51:23.215 create-user
2018/01/25 20:51:23.585
saving user [theburningmonk] in the [user] table…
saved user [theburningmonk] in the [user] table
level
debug
debug
debug uploading profile image…
create-user debug tagged user [theburningmonk] with Azure Face API…
create-user2018/01/25 20:51:23.587
create-auth0-user

159. user
profile-images
POST /user
process-images
resize-images
image-tasks
Auth0
create-user
reformat-imagestag-user
Face API
Logs
timestamp component message
2018/01/25 20:51:23.201 create-user
2018/01/25 20:51:23.215 create-user
2018/01/25 20:51:23.585
saving user [theburningmonk] in the [user] table…
saved user [theburningmonk] in the [user] table
level
debug
debug
debug uploading profile image…
create-user debug tagged user [theburningmonk] with Azure Face API…
create-user2018/01/25 20:51:23.587
click here to go to code create-auth0-user

161. Logs Input/Output
user
profile-images
POST /user
process-images
resize-images
image-tasks
Auth0
create-user
reformat-imagestag-user
Face API
input output
{
"body": "{ "username":"theburningmonk"}",
"resource": "/user",
"requestContext": {
"resourceId": "123456",
"apiId": “1234567890",
"resourcePath": "/user",
{
"statusCode": 200
}
create-auth0-user

162. Logs Input/Output
user
profile-images
POST /user
process-images
resize-images
image-tasks
Auth0
create-user
reformat-imagestag-user
Face API
input output
{ "Records": [
{ "Sns": {
"Type": "Notification",
"MessageId": "…",
"TopicArn": "…",
"Message": "…",
"Timestamp": "2018/01/25 20:51:24.215",
{
"error": null,
"result": "OK"
}
create-auth0-user

163. Logs Input/Output
user
profile-images
POST /user
process-images
resize-images
image-tasks
Auth0
create-user
reformat-imagestag-user
Face API
input error
{ "Records": [
{ "Sns": {
"Type": "Notification",
"MessageId": "…",
"TopicArn": "…",
"Message": "…",
"Timestamp": "2018/01/25 20:51:24.215",
[com.spaceape.dragon.handler.ReformatProfileImageHandle
r] Null reference exception
*java.lang.NullReferenceException: …
* at …
* at …
* at …
create-auth0-user

164. Logs Input/Output
user
profile-images
POST /user
process-images
resize-images
image-tasks
Auth0
create-user
create-auth0-user
reformat-imagestag-user
Face API
input error
{ "Records": [
{ "Sns": {
"Type": "Notification",
"MessageId": "…",
"TopicArn": "…",
"Message": "…",
"Timestamp": "2018/01/25 20:51:24.215",
[com.spaceape.dragon.handler.ReformatProfileImageHandle
r] Null reference exception
*java.lang.NullReferenceException: …
* at …
* at …
* at …
!

166. All
0 200 400 600 800
create-user
…user.insert_user
…user.upload_img
tag-user
create-auto0-user
process-images
resize-images
reformat-images!
837ms
406ms
66ms
114ms
122ms
82ms
240ms
157ms
35ms

167. All
0 200 400 600 800
create-user
…user.insert_user
…user.upload_img
tag-user
create-auto0-user
process-images
resize-images
reformat-images!
837ms
406ms
66ms
114ms
122ms
82ms
240ms
157ms
35ms

168. Input/Output
user
profile-images
POST /user
process-images
resize-images
image-tasks
Auth0
create-user
create-auth0-user
reformat-imagestag-user
Face API
Logs
!
All
0 200 400 600 800
create-user
…user.insert_user
…user.upload_img
tag-user
create-auto0-user
process-images
resize-images
reformat-images!
837ms
406ms
66ms
114ms
122ms
82ms
240ms
157ms
35ms

169. Input/Output
user
profile-images
POST /user
process-images
resize-images
image-tasks
Auth0
create-user
create-auth0-user
reformat-imagestag-user
Face API
Logs
!
All
0 200 400 600 800
create-user
…user.insert_user
…user.upload_img
tag-user
create-auto0-user
process-images
resize-images
reformat-images!
837ms
406ms
66ms
114ms
122ms
82ms
240ms
157ms
35ms

170. all your needs in one placeTRACING

172. mmm… it’s a graph

173. what if we can query it
like a graph?

175. http://amzn.to/2nk7uiW

176. ability to query based on the relationship
between observed components
(as well as the components themselves)

177. root cause analysis

178. the elevated error rate in service X was caused by
DynamoDB table throttling.“
”

179. payment was slow last
night around 10PM.
investigate.

180. time
95-percentile latency
service A
service B
10PM

181. time
95-percentile latency
service A
service B
10PM
causality? or correlation?

182. user-service
USESUSES
DEPENDS_ON
auth-serviceUSES
payment-service
DEPENDS_ON
“payment was slow last
night around 10PM”
user-table

183. user-service
USESUSES
DEPENDS_ON
auth-serviceUSES
DEPENDS_ON
payment-service
user-table
throttled exceptions!

184. user-table
user-stream
DEPENDS_ON
DEPENDS_ON USES
USES
USES
USES
USES
DEPENDS_ON
D
EPEN
D
S_O
N
DEPENDS_ON
PUBLISHES_TO
“what else is impacted by the throttled exceptions on user-table?”

185. user-table
user-stream
DEPENDS_ON
DEPENDS_ON USES
USES
USES
USES
USES
DEPENDS_ON
D
EPEN
D
S_O
N
DEPENDS_ON
PUBLISHES_TO
“what else is impacted by the throttled exceptions on user-table?”

186. wouldn’t that be nice?

188. MACHINE
LEARNING

189. use ML to auto-detect erroneous or
suspicious behaviours, or to suggest
possible improvements

191. !
Function [X] just performed
an unexpected write against
DynamoDB table [Y].
Should I…
ignore it from now on
shut it down!!

192. Observability Bot <bot@bestobservability.com>

193. Observability Bot <bot@bestobservability.com>
don’t bother me about this again

194. Observability Bot <bot@bestobservability.com>
auto-modify IAM role with DENY rule

195. Function [X]’s performance
has degraded since yesterday -
99% latency has gone up by
47% from 100ms to 147ms.
!

196. !
Function [X] can run faster &
cheaper if you increase its
memory allocation.
Should I…
ignore it from now on
update setting

197. zzz… the future of… zzz …
serverless observability… zzz

199. Simon Wardley

200. Simon Wardley
context &
movement

201. However, I would argue that the health of the system no
longer matters. We've entered an era where what matters is
the health of each individual event, or each individual user's
experience, or each shopping cart's experience (or other high
cardinality dimensions). With distributed systems you don't
care about the health of the system, you care about the
health of the event or the slice.
”http://bit.ly/2E2QngU- Charity Majors
“

202. “one user action/vertical slice through the system”

203. movement
context
movement

204. The best way to predict the future
is to invent it.
- Alan Kay

205. The best way to invent
the future is to inception
someone else to do it.
- me