The document discusses the importance of observability in serverless architectures, highlighting the need to focus on the health of individual events rather than overall system health. It outlines key components of observability, including monitoring, visualization, and tracing in distributed systems, and presents challenges associated with implementing observability due to the nature of serverless computing. Additionally, it emphasizes the need for improved tools to understand complex interactions in microservices and asynchronous processes.

1. and FUTUREThe
Serverless
OBSERVABILITY
ofpresent

2. InfoQ.com: News & Community Site
Watch the video with slide
synchronization on InfoQ.com!
https://www.infoq.com/presentations/
serverless-observability
• Over 1,000,000 software developers, architects and CTOs read the site world-
wide every month
• 250,000 senior developers subscribe to our weekly newsletter
• Published in 4 languages (English, Chinese, Japanese and Brazilian
Portuguese)
• Post content from our QCon conferences
• 2 dedicated podcast channels: The InfoQ Podcast, with a focus on
Architecture and The Engineering Culture Podcast, with a focus on building
• 96 deep dives on innovative topics packed as downloadable emags and
minibooks
• Over 40 new content items per week

3. Purpose of QCon
- to empower software development by facilitating the spread of
knowledge and innovation
Strategy
- practitioner-driven conference designed for YOU: influencers of
change and innovation in your teams
- speakers and topics driving the evolution and innovation
- connecting and catalyzing the influencers and innovators
Highlights
- attended by more than 12,000 delegates since 2007
- held in 9 cities worldwide
Presented at QCon London
www.qconlondon.com

4. hi, my name is Yan.

5. hi, my name is Yan.

6. hi, my name is Yan.
AWS user since 2009

7. http://bit.ly/yubl-serverless

8. http://bit.ly/2Cdsai5

9. 2017
observability

10. http://bit.ly/2EXQZBj

11. http://bit.ly/2EXKEFZ

12. mm… I wonder what’s
going on here…

13. what is observability?
how is it different from monitoring?

14. Monitoring
watching out for
known failure modes
in the system,
e.g. network I/O, CPU,
memory usage, …

15. Observability
being able to debug
the system, and gain
insights into the
system’s behaviour

16. However, I would argue that the health of the system no
longer matters. We've entered an era where what matters is
the health of each individual event, or each individual user's
experience, or each shopping cart's experience (or other high
cardinality dimensions). With distributed systems you
don't care about the health of the system, you care about
the health of the event or the slice.
”http://bit.ly/2E2QngU- Charity Majors
“

17. However, I would argue that the health of the system no
longer matters. We've entered an era where what matters is
the health of each individual event, or each individual user's
experience, or each shopping cart's experience (or other high
cardinality dimensions). With distributed systems you
don't care about the health of the system, you care about
the health of the event or the slice.
”http://bit.ly/2E2QngU- Charity Majors
“

18. These are the four pillars of the Observability Engineering
team’s charter:
• Monitoring
• Alerting/visualization
• Distributed systems tracing infrastructure
• Log aggregation/analytics
“
” http://bit.ly/2DnjyuW- Observability Engineering at Twitter

19. Observability is useful even outside of incidents and outages

20. microservices death stars circa 2015

25. microservices death stars circa 2015
I got this!

27. new
challenges

28. new
challenges

29. NO ACCESS
to underlying OS

30. NOWHERE
to install agents/daemons

31. •nowhere to install agents/daemons
new challenges

32. user request
user request
user request
user request
user request
user request
user request
critical paths:
minimise user-facing latency
handler
handler
handler
handler
handler
handler
handler

33. user request
user request
user request
user request
user request
user request
user request
critical paths:
minimise user-facing latency
StatsD
handler
handler
handler
handler
handler
handler
handler
rsyslog
background processing:
batched, asynchronous, low
overhead

34. user request
user request
user request
user request
user request
user request
user request
critical paths:
minimise user-facing latency
StatsD
handler
handler
handler
handler
handler
handler
handler
rsyslog
background processing:
batched, asynchronous, low
overhead
NO background processing
except what platform provides

35. •no background processing
•nowhere to install agents/daemons
new challenges

36. EC2
concurrency used to be
handled by your code

37. EC2
Lambda
Lambda
Lambda
Lambda
Lambda
now, it’s handled by the
AWS Lambda platform

38. EC2
logs & metrics used to be
batched here

39. EC2
Lambda
Lambda
Lambda
Lambda
Lambda
now, they are batched in each
concurrent execution, at best…

42. HIGHER concurrency to log
aggregation/telemetry system

43. •higher concurrency to telemetry system
•nowhere to install agents/daemons
•no background processing
new challenges

44. Lambda
cold start

45. Lambda
data is batched between
invocations

46. Lambda
idle
data is batched between
invocations

47. Lambda
idle
garbage collectiondata is batched between
invocations

48. Lambda
idle
garbage collectiondata is batched between
invocations
HIGH chance of data loss

49. •high chance of data loss (if batching)
•nowhere to install agents/daemons
•no background processing
•higher concurrency to telemetry system
new challenges

50. Lambda

51. my code
send metrics

52. my code
send metrics

53. my code
send metrics
internet internet
press button something happens

55. http://bit.ly/2Dpidje

56. ?
functions are often chained together
via asynchronous invocations

57. ?
SNS
Kinesis
CloudWatch
Events
CloudWatch
LogsIoT
DynamoDB
S3 SES

58. ?
SNS
Kinesis
CloudWatch
Events
CloudWatch
LogsIoT
DynamoDB
S3 SES
tracing ASYNCHRONOUS
invocations through so many
different event sources is difficult

59. •asynchronous invocations
•nowhere to install agents/daemons
•no background processing
•higher concurrency to telemetry system
•high chance of data loss (if batching)
new challenges

60. the Present

61. These are the four pillars of the Observability Engineering
team’s charter:
• Monitoring
• Alerting/visualization
• Distributed systems tracing infrastructure
• Log aggregation/analytics
“
” http://bit.ly/2DnjyuW- Observability Engineering at Twitter

64. 2016-07-12T12:24:37.571Z 994f18f9-482b-11e6-8668-53e4eab441ae
GOT is off air, what do I do now?

65. 2016-07-12T12:24:37.571Z 994f18f9-482b-11e6-8668-53e4eab441ae
GOT is off air, what do I do now?
UTC Timestamp Request Id
your log message

67. one log group per
function
one log stream for each
concurrent invocation

68. logs are not easily searchable in
CloudWatch Logs
me

69. CloudWatch Logs

70. CloudWatch Logs AWS Lambda ELK stack

71. …

72. CloudWatch Logs

76. •no background processing
•nowhere to install agents/daemons
new challenges

77. my code
send metrics
internet internet
press button something happens

78. those extra 10-20ms for
sending custom metrics would
compound when you have
microservices and multiple
APIs are called within one slice
of user event

79. Amazon found every 100ms of latency cost them 1% in sales.
http://bit.ly/2EXPfbA

80. console.log(“hydrating yubls from db…”);
console.log(“fetching user info from user-api”);
console.log(“MONITORING|1489795335|27.4|latency|user-api-latency”);
console.log(“MONITORING|1489795335|8|count|yubls-served”);
timestamp metric value
metric type
metric namemetrics
logs

81. CloudWatch Logs AWS Lambda
ELK stack
logs
m
etrics
CloudWatch

83. delay
cost
concurrency

84. delay
cost
concurrency
no latency
overhead

85. API Gateway
send custom metrics
asynchronously

86. SNS KinesisS3API Gateway
…
send custom metrics
asynchronously
send custom metrics as
part of function invocation

88. X-Ray

91. do not span over API Gateway

92. narrow focus on a function
good for homing in on performance issues
for a particular function, but offers little to
help you build intuition about how your
system operates as a whole.

93. However, I would argue that the health of the system no
longer matters. We've entered an era where what matters is
the health of each individual event, or each individual user's
experience, or each shopping cart's experience (or other high
cardinality dimensions). With distributed systems you don't
care about the health of the system, you care about the
health of the event or the slice.
”http://bit.ly/2E2QngU- Charity Majors
“

94. follow the data

96. don’t span over async invocations
good for identifying dependencies of a function,
but not good enough for tracing the entire call
chain as user request/data flows through the
system via async event sources.

97. don’t span over non-AWS services

99. static view

100. our tools need to do more to help us with
understanding & debugging our distributed system,
not just what happens inside one function

102. “one user action/vertical slice through the system”

103. microservices death stars circa 2015

104. microservices death stars circa 2015
HELP…

106. WARNING: this is part fiction, part inspired by new tools

107. DASHBOARDS

111. different dimensions of X splattered
across the screen

114. + cold starts
+ throttled invocations
+ concurrent executions
+ estimated cost ($)

115. SubscriberGetAccount
200,545
0
19
94
0
0 %
0 %
Est Cost:
Req Rate:
$54.0/s
20,056.0/s
Concurrency
Median
Mean 99.5th
99th
90th370
1ms
4ms 61ms
44ms
10ms
circle colour and size represent
health and traffic volume
2 minutes of request rate to
show relative changes in traffic
no. of concurrent executions
of this function
Request rate
Estimated cost
Error percentage
of last 10 seconds
Cold start percentage
last 10 seconds
last minute latency percentiles
200,545
0
19
94
0
Rolling 10 second counters
with 1 second granularity
Successes
Cold starts
Timeouts
Throttled Invocations
Errors

116. SubscriberGetAccount
200,545
0
19
94
0
0 %
0 %
Est Cost:
Req Rate:
$54.0/s
20,056.0/s
Concurrency
Median
Mean 99.5th
99th
90th370
1ms
4ms 61ms
44ms
10ms
circle colour and size represent
health and traffic volume
2 minutes of request rate to
show relative changes in traffic
no. of concurrent executions
of this function
Request rate
Estimated cost
Error percentage
of last 10 seconds
Cold start percentage
last 10 seconds
last minute latency percentiles
200,545
0
19
94
0
Rolling 10 second counters
with 1 second granularity
Successes
Cold starts
Timeouts
Throttled Invocations
Errors

117. SubscriberGetAccount
200,545
0
19
94
0
0 %
0 %
Est Cost:
Req Rate:
$54.0/s
20,056.0/s
Concurrency
Median
Mean 99.5th
99th
90th370
1ms
4ms 61ms
44ms
10ms
circle colour and size represent
health and traffic volume
2 minutes of request rate to
show relative changes in traffic
no. of concurrent executions
of this function
Request rate
Estimated cost
Error percentage
of last 10 seconds
Cold start percentage
last 10 seconds
last minute latency percentiles
200,545
0
19
94
0
Rolling 10 second counters
with 1 second granularity
Successes
Cold starts
Timeouts
Throttled Invocations
Errors

118. SubscriberGetAccount
200,545
0
19
94
0
0 %
0 %
Est Cost:
Req Rate:
$54.0/s
20,056.0/s
Concurrency
Median
Mean 99.5th
99th
90th370
1ms
4ms 61ms
44ms
10ms
circle colour and size represent
health and traffic volume
2 minutes of request rate to
show relative changes in traffic
no. of concurrent executions
of this function
Request rate
Estimated cost
Error percentage
of last 10 seconds
Cold start percentage
last 10 seconds
last minute latency percentiles
200,545
0
19
94
0
Rolling 10 second counters
with 1 second granularity
Successes
Cold starts
Timeouts
Throttled Invocations
Errors

119. SubscriberGetAccount
200,545
0
19
94
0
0 %
0 %
Est Cost:
Req Rate:
$54.0/s
20,056.0/s
Concurrency
Median
Mean 99.5th
99th
90th370
1ms
4ms 61ms
44ms
10ms
circle colour and size represent
health and traffic volume
2 minutes of request rate to
show relative changes in traffic
no. of concurrent executions
of this function
Request rate
Estimated cost
Error percentage
of last 10 seconds
Cold start percentage
last 10 seconds
last minute latency percentiles
200,545
0
19
94
0
Rolling 10 second counters
with 1 second granularity
Successes
Cold starts
Timeouts
Throttled Invocations
Errors

125. birds-eye view of our system as it lives and breathes

127. user
profile-images
POST /user
process-images
resize-images
image-tasks
Auth0
create-user
reformat-imagestag-user
Face API
create-auth0-user

128. user
profile-images
POST /user
process-images
resize-images
image-tasks
Auth0
create-user
reformat-imagestag-user
Face API
trace async invocations
create-auth0-user

129. user
profile-images
POST /user
process-images
resize-images
image-tasks
Auth0
create-user
reformat-imagestag-user
Face API
trace non-AWS resources
create-auth0-user

130. user
profile-images
POST /user
process-images
resize-images
image-tasks
Auth0
create-user
reformat-imagestag-user
Face API
Logs
timestamp component message
POST /user2018/01/25 20:51:23.188
2018/01/25 20:51:23.201 create-user
2018/01/25 20:51:23.215 create-user
2018/01/25 20:51:23.521 tag-user
incoming request…
saving user [theburningmonk] in the [user] table…
saved user [theburningmonk] in the [user] table
level
debug
debug
debug
debug tagging user [theburningmonk] with Azure Face API…
create-auth0-user

131. user
profile-images
POST /user
process-images
resize-images
image-tasks
Auth0
create-user
reformat-imagestag-user
Face API
Logs
timestamp component message
POST /user2018/01/25 20:51:23.188
2018/01/25 20:51:23.201 create-user
2018/01/25 20:51:23.215 create-user
2018/01/25 20:51:23.521 tag-user
incoming request…
saving user [theburningmonk] in the [user] table…
saved user [theburningmonk] in the [user] table
level
debug
debug
debug
debug tagging user [theburningmonk] with Azure Face API…
create-auth0-user

132. user
profile-images
POST /user
process-images
resize-images
image-tasks
Auth0
create-user
reformat-imagestag-user
Face API
Logs
timestamp component message
POST /user2018/01/25 20:51:23.188 incoming request…
level
debug
request-id
start-time
0ae4ba5d-dab1-4f9e-9de7-eace27ebfbc2
2018/01/25 20:51:23.188
method POST
create-auth0-user

133. user
profile-images
POST /user
process-images
resize-images
image-tasks
Auth0
create-user
reformat-imagestag-user
Face API
Logs
timestamp component message
2018/01/25 20:51:23.201 create-user
2018/01/25 20:51:23.215 create-user
2018/01/25 20:51:23.585
saving user [theburningmonk] in the [user] table…
saved user [theburningmonk] in the [user] table
level
debug
debug
debug uploading profile image…
create-user debug tagged user [theburningmonk] with Azure Face API…
create-user2018/01/25 20:51:23.587
create-auth0-user

134. user
profile-images
POST /user
process-images
resize-images
image-tasks
Auth0
create-user
reformat-imagestag-user
Face API
Logs
timestamp component message
2018/01/25 20:51:23.201 create-user
2018/01/25 20:51:23.215 create-user
2018/01/25 20:51:23.585
saving user [theburningmonk] in the [user] table…
saved user [theburningmonk] in the [user] table
level
debug
debug
debug uploading profile image…
create-user debug tagged user [theburningmonk] with Azure Face API…
create-user2018/01/25 20:51:23.587
click here to go to code create-auth0-user

136. Logs Input/Output
user
profile-images
POST /user
process-images
resize-images
image-tasks
Auth0
create-user
reformat-imagestag-user
Face API
input output
{
"body": "{ "username":"theburningmonk"}",
"resource": "/user",
"requestContext": {
"resourceId": "123456",
"apiId": “1234567890",
"resourcePath": "/user",
{
"statusCode": 200
}
create-auth0-user

137. Logs Input/Output
user
profile-images
POST /user
process-images
resize-images
image-tasks
Auth0
create-user
reformat-imagestag-user
Face API
input output
{ "Records": [
{ "Sns": {
"Type": "Notification",
"MessageId": "…",
"TopicArn": "…",
"Message": "…",
"Timestamp": "2018/01/25 20:51:24.215",
{
"error": null,
"result": "OK"
}
create-auth0-user

138. Logs Input/Output
user
profile-images
POST /user
process-images
resize-images
image-tasks
Auth0
create-user
reformat-imagestag-user
Face API
input error
{ "Records": [
{ "Sns": {
"Type": "Notification",
"MessageId": "…",
"TopicArn": "…",
"Message": "…",
"Timestamp": "2018/01/25 20:51:24.215",
[com.spaceape.dragon.handler.ReformatProfileImageHandle
r] Null reference exception
*java.lang.NullReferenceException: …
* at …
* at …
* at …
create-auth0-user

139. Logs Input/Output
user
profile-images
POST /user
process-images
resize-images
image-tasks
Auth0
create-user
create-auth0-user
reformat-imagestag-user
Face API
input error
{ "Records": [
{ "Sns": {
"Type": "Notification",
"MessageId": "…",
"TopicArn": "…",
"Message": "…",
"Timestamp": "2018/01/25 20:51:24.215",
[com.spaceape.dragon.handler.ReformatProfileImageHandle
r] Null reference exception
*java.lang.NullReferenceException: …
* at …
* at …
* at …
!

141. All
0 200 400 600 800
create-user
…user.insert_user
…user.upload_img
tag-user
create-auto0-user
process-images
resize-images
reformat-images!
837ms
406ms
66ms
114ms
122ms
82ms
240ms
157ms
35ms

142. All
0 200 400 600 800
create-user
…user.insert_user
…user.upload_img
tag-user
create-auto0-user
process-images
resize-images
reformat-images!
837ms
406ms
66ms
114ms
122ms
82ms
240ms
157ms
35ms

143. Input/Output
user
profile-images
POST /user
process-images
resize-images
image-tasks
Auth0
create-user
create-auth0-user
reformat-imagestag-user
Face API
Logs
!
All
0 200 400 600 800
create-user
…user.insert_user
…user.upload_img
tag-user
create-auto0-user
process-images
resize-images
reformat-images!
837ms
406ms
66ms
114ms
122ms
82ms
240ms
157ms
35ms

144. Input/Output
user
profile-images
POST /user
process-images
resize-images
image-tasks
Auth0
create-user
create-auth0-user
reformat-imagestag-user
Face API
Logs
!
All
0 200 400 600 800
create-user
…user.insert_user
…user.upload_img
tag-user
create-auto0-user
process-images
resize-images
reformat-images!
837ms
406ms
66ms
114ms
122ms
82ms
240ms
157ms
35ms

145. all your needs in one placeTRACING

147. mmm… it’s a graph

148. what if we can query it
like a graph?

150. http://amzn.to/2nk7uiW

151. ability to query based on the relationship
between observed components
(as well as the components themselves)

152. root cause analysis

153. the elevated error rate in service X was caused by
DynamoDB table throttling.“
”

154. payment was slow last
night around 10PM.
investigate.

155. time
95-percentile latency
service A
service B
10PM

156. time
95-percentile latency
service A
service B
10PM
causality? or correlation?

157. user-service
USESUSES
DEPENDS_ON
auth-serviceUSES
payment-service
DEPENDS_ON
“payment was slow last
night around 10PM”
user-table

158. user-service
USESUSES
DEPENDS_ON
auth-serviceUSES
DEPENDS_ON
payment-service
user-table
throttled exceptions!

159. user-table
user-stream
DEPENDS_ON
DEPENDS_ON USES
USES
USES
USES
USES
DEPENDS_ON
D
EPEN
D
S_O
N
DEPENDS_ON
PUBLISHES_TO
“what else is impacted by the throttled exceptions on user-table?”

160. user-table
user-stream
DEPENDS_ON
DEPENDS_ON USES
USES
USES
USES
USES
DEPENDS_ON
D
EPEN
D
S_O
N
DEPENDS_ON
PUBLISHES_TO
“what else is impacted by the throttled exceptions on user-table?”

161. wouldn’t that be nice?

163. MACHINE
LEARNING

164. use ML to auto-detect erroneous or
suspicious behaviours, or to suggest
possible improvements

166. !
Function [X] just performed
an unexpected write against
DynamoDB table [Y].
Should I…
ignore it from now on
shut it down!!

167. Observability Bot <bot@bestobservability.com>

168. Observability Bot <bot@bestobservability.com>
don’t bother me about this again

169. Observability Bot <bot@bestobservability.com>
auto-modify IAM role with DENY rule

170. Function [X]’s performance
has degraded since yesterday -
99% latency has gone up by
47% from 100ms to 147ms.
!

171. !
Function [X] can run faster &
cheaper if you increase its
memory allocation.
Should I…
ignore it from now on
update setting

172. zzz… the future of… zzz …
serverless observability… zzz

174. Simon Wardley

175. Simon Wardley
context &
movement

176. However, I would argue that the health of the system no
longer matters. We've entered an era where what matters is
the health of each individual event, or each individual user's
experience, or each shopping cart's experience (or other high
cardinality dimensions). With distributed systems you don't
care about the health of the system, you care about the
health of the event or the slice.
”http://bit.ly/2E2QngU- Charity Majors
“

177. “one user action/vertical slice through the system”

178. movement
context
movement

179. The best way to predict the future
is to invent it.
- Alan Kay

180. The best way to invent
the future is to inception
someone else to do it.
- me

181. Serkan Özal
@serkan_ozal

182. Nitzan Shapira
@nitzanshapira
Ran Ribenzaft
@ranrib

183. Adam Johnson
@adjohn
Erica Windisch
@ewindisch

184. Charity Majors
@mipsytipsy
Cindy Sridharan
@copyconstruct
Erica Windisch
@ewindisch
Liz Fong-Jones
@lizthegrey
JBD
@rakyll

185. API Gateway and Kinesis
Authentication & authorisation (IAM, Cognito)
Testing
Running & Debugging functions locally
Log aggregation
Monitoring & Alerting
X-Ray
Correlation IDs
CI/CD
Performance and Cost optimisation
Error Handling
Configuration management
VPC
Security
Leading practices (API Gateway, Kinesis, Lambda)
Canary deployments
http://bit.ly/production-ready-serverless
get 40% off
with code:
ytcui

186. Watch the video with slide
synchronization on InfoQ.com!
https://www.infoq.com/presentations/
serverless-observability