1. and FUTUREThe
Serverless
OBSERVABILITY
ofpresent

2. hi, my name is Yan.

3. I’m a principal engineer at

6. available in:
Austria, Switzerland,
Germany, Japan and
Canada
30+ platforms

7. coming to the US

8. follow @DAZN_ngnrs
for updates about the
engineering team
We’re hiring! Visit
engineering.dazn.com
to learn more.

9. AWS user since 2009

10. http://bit.ly/yubl-serverless

11. http://bit.ly/2Cdsai5

12. 2017
observability

13. Monitoring
watching out for
known failure modes
in the system,
e.g. network I/O, CPU,
memory usage, …

14. Observability
being able to debug
the system, and gain
insights into the
system’s behaviour

15. http://bit.ly/2EXKEFZ

16. mm… I wonder what’s
going on here…

17. These are the four pillars of the Observability Engineering
team’s charter:
• Monitoring
• Alerting/visualization
• Distributed systems tracing infrastructure
• Log aggregation/analytics
“
” http://bit.ly/2DnjyuW- Observability Engineering at Twitter

18. microservices death stars circa 2015

23. microservices death stars circa 2015
I got this!

25. new
challenges

26. new
challenges

27. NO ACCESS
to underlying OS

28. NOWHERE
to install agents/daemons

29. •nowhere to install agents/daemons
new challenges

30. user request
user request
user request
user request
user request
user request
user request
critical paths:
minimise user-facing latency
handler
handler
handler
handler
handler
handler
handler

31. user request
user request
user request
user request
user request
user request
user request
critical paths:
minimise user-facing latency
StatsD
handler
handler
handler
handler
handler
handler
handler
rsyslog
background processing:
batched, asynchronous, low
overhead

32. user request
user request
user request
user request
user request
user request
user request
critical paths:
minimise user-facing latency
StatsD
handler
handler
handler
handler
handler
handler
handler
rsyslog
background processing:
batched, asynchronous, low
overhead
NO background processing
except what platform provides

33. •no background processing
•nowhere to install agents/daemons
new challenges

34. EC2
concurrency used to be
handled by your code

35. EC2
Lambda
Lambda
Lambda
Lambda
Lambda
now, it’s handled by the
AWS Lambda platform

36. EC2
logs & metrics used to be
batched here

37. EC2
Lambda
Lambda
Lambda
Lambda
Lambda
now, they are batched in each
concurrent execution, at best…

40. HIGHER concurrency to log
aggregation/telemetry system

41. •higher concurrency to telemetry system
•nowhere to install agents/daemons
•no background processing
new challenges

42. Lambda
cold start

43. Lambda
data is batched between
invocations

44. Lambda
idle
data is batched between
invocations

45. Lambda
idle
garbage collectiondata is batched between
invocations

46. Lambda
idle
garbage collectiondata is batched between
invocations
HIGH chance of data loss

47. •high chance of data loss (if batching)
•nowhere to install agents/daemons
•no background processing
•higher concurrency to telemetry system
new challenges

48. Lambda

49. my code
send metrics

50. my code
send metrics

51. my code
send metrics
internet internet
press button something happens

53. http://bit.ly/2Dpidje

54. ?
functions are often chained together
via asynchronous invocations

55. ?
SNS
Kinesis
CloudWatch
Events
CloudWatch
LogsIoT
DynamoDB
S3 SES

56. ?
SNS
Kinesis
CloudWatch
Events
CloudWatch
LogsIoT
DynamoDB
S3 SES
tracing ASYNCHRONOUS
invocations through so many
different event sources is difficult

57. •asynchronous invocations
•nowhere to install agents/daemons
•no background processing
•higher concurrency to telemetry system
•high chance of data loss (if batching)
new challenges

58. the Present

59. These are the four pillars of the Observability Engineering
team’s charter:
• Monitoring
• Alerting/visualization
• Distributed systems tracing infrastructure
• Log aggregation/analytics
“
” http://bit.ly/2DnjyuW- Observability Engineering at Twitter

62. 2016-07-12T12:24:37.571Z 994f18f9-482b-11e6-8668-53e4eab441ae
GOT is off air, what do I do now?

63. 2016-07-12T12:24:37.571Z 994f18f9-482b-11e6-8668-53e4eab441ae
GOT is off air, what do I do now?
UTC Timestamp Request Id
your log message

65. one log group per
function
one log stream for each
concurrent invocation

66. logs are not easily searchable in
CloudWatch Logs
me

67. CloudWatch Logs

68. CloudWatch Logs AWS Lambda ELK stack

69. …

70. CloudWatch Logs

76. CloudWatch Logs

78. •no background processing
•nowhere to install agents/daemons
new challenges

79. my code
send metrics
internet internet
press button something happens

80. those extra 10-20ms for
sending custom metrics would
compound when you have
microservices and multiple
APIs are called within one slice
of user event

81. Amazon found every 100ms of latency cost them 1% in sales.
http://bit.ly/2EXPfbA

82. console.log(“hydrating yubls from db…”);
console.log(“fetching user info from user-api”);
console.log(“MONITORING|1489795335|27.4|latency|user-api-latency”);
console.log(“MONITORING|1489795335|8|count|yubls-served”);
timestamp metric value
metric type
metric namemetrics
logs

83. CloudWatch Logs AWS Lambda
ELK stack
logs
m
etrics
CloudWatch

85. delay
cost
concurrency

86. delay
cost
concurrency
no latency
overhead

87. API Gateway
send custom metrics
asynchronously

88. SNS KinesisS3API Gateway
…
send custom metrics
asynchronously
send custom metrics as
part of function invocation

90. X-Ray

93. do not span over API Gateway

94. narrow focus on a function
good for homing in on performance issues
for a particular function, but offers little to
help you build intuition about how your
system operates as a whole.

95. However, I would argue that the health of the system no
longer matters. We've entered an era where what matters is
the health of each individual event, or each individual user's
experience, or each shopping cart's experience (or other high
cardinality dimensions). With distributed systems you don't
care about the health of the system, you care about the
health of the event or the slice.
”http://bit.ly/2E2QngU- Charity Majors
“

96. follow the data

98. don’t span over async
invocations
good for identifying dependencies of a function,
but not good enough for tracing the entire call
chain as user request/data flows through the
system via async event sources.

99. don’t span over non-AWS services

101. static view

102. our tools need to do more to help us with
understanding & debugging our distributed
system, not just what happens inside one function

104. “one user action/vertical slice through the system”

105. microservices death stars circa 2015

106. microservices death stars circa 2015
HELP…

108. WARNING: this is part fiction, part inspired by new tools

109. DASHBOARDS

113. different dimensions of service X
splattered across the screen

116. +cold starts
+throttled invocations
+concurrent executions
+estimated cost ($)

117. SubscriberGetAccount
200,545
0
19
94
0
0 %
0 %
Est Cost:
Req Rate:
$54.0/s
20,056.0/s
Concurrency
Median
Mean 99.5th
99th
90th370
1ms
4ms 61ms
44ms
10ms
circle colour and size represent
health and traffic volume
2 minutes of request rate to
show relative changes in traffic
no. of concurrent executions
of this function
Request rate
Estimated cost
Error percentage
of last 10 seconds
Cold start percentage
last 10 seconds
last minute latency percentiles
200,545
0
19
94
0
Rolling 10 second counters
with 1 second granularity
Successes
Cold starts
Timeouts
Throttled Invocations
Errors

118. SubscriberGetAccount
200,545
0
19
94
0
0 %
0 %
Est Cost:
Req Rate:
$54.0/s
20,056.0/s
Concurrency
Median
Mean 99.5th
99th
90th370
1ms
4ms 61ms
44ms
10ms
circle colour and size represent
health and traffic volume
2 minutes of request rate to
show relative changes in traffic
no. of concurrent executions
of this function
Request rate
Estimated cost
Error percentage
of last 10 seconds
Cold start percentage
last 10 seconds
last minute latency percentiles
200,545
0
19
94
0
Rolling 10 second counters
with 1 second granularity
Successes
Cold starts
Timeouts
Throttled Invocations
Errors

119. SubscriberGetAccount
200,545
0
19
94
0
0 %
0 %
Est Cost:
Req Rate:
$54.0/s
20,056.0/s
Concurrency
Median
Mean 99.5th
99th
90th370
1ms
4ms 61ms
44ms
10ms
circle colour and size represent
health and traffic volume
2 minutes of request rate to
show relative changes in traffic
no. of concurrent executions
of this function
Request rate
Estimated cost
Error percentage
of last 10 seconds
Cold start percentage
last 10 seconds
last minute latency percentiles
200,545
0
19
94
0
Rolling 10 second counters
with 1 second granularity
Successes
Cold starts
Timeouts
Throttled Invocations
Errors

120. SubscriberGetAccount
200,545
0
19
94
0
0 %
0 %
Est Cost:
Req Rate:
$54.0/s
20,056.0/s
Concurrency
Median
Mean 99.5th
99th
90th370
1ms
4ms 61ms
44ms
10ms
circle colour and size represent
health and traffic volume
2 minutes of request rate to
show relative changes in traffic
no. of concurrent executions
of this function
Request rate
Estimated cost
Error percentage
of last 10 seconds
Cold start percentage
last 10 seconds
last minute latency percentiles
200,545
0
19
94
0
Rolling 10 second counters
with 1 second granularity
Successes
Cold starts
Timeouts
Throttled Invocations
Errors

121. SubscriberGetAccount
200,545
0
19
94
0
0 %
0 %
Est Cost:
Req Rate:
$54.0/s
20,056.0/s
Concurrency
Median
Mean 99.5th
99th
90th370
1ms
4ms 61ms
44ms
10ms
circle colour and size represent
health and traffic volume
2 minutes of request rate to
show relative changes in traffic
no. of concurrent executions
of this function
Request rate
Estimated cost
Error percentage
of last 10 seconds
Cold start percentage
last 10 seconds
last minute latency percentiles
200,545
0
19
94
0
Rolling 10 second counters
with 1 second granularity
Successes
Cold starts
Timeouts
Throttled Invocations
Errors

127. birds-eye view of our system as it lives and breathes

129. user
profile-images
POST /user
process-images
resize-images
image-tasks
Auth0
create-user
reformat-imagestag-user
Face API
create-auth0-user

130. user
profile-images
POST /user
process-images
resize-images
image-tasks
Auth0
create-user
reformat-imagestag-user
Face API
trace async invocations
create-auth0-user

131. user
profile-images
POST /user
process-images
resize-images
image-tasks
Auth0
create-user
reformat-imagestag-user
Face API
trace non-AWS resources
create-auth0-user

132. user
profile-images
POST /user
process-images
resize-images
image-tasks
Auth0
create-user
reformat-imagestag-user
Face API
Logs
timestamp component message
POST /user2018/01/25 20:51:23.188
2018/01/25 20:51:23.201 create-user
2018/01/25 20:51:23.215 create-user
2018/01/25 20:51:23.521 tag-user
incoming request…
saving user [theburningmonk] in the [user] table…
saved user [theburningmonk] in the [user] table
level
debug
debug
debug
debug tagging user [theburningmonk] with Azure Face API…
create-auth0-user

133. user
profile-images
POST /user
process-images
resize-images
image-tasks
Auth0
create-user
reformat-imagestag-user
Face API
Logs
timestamp component message
POST /user2018/01/25 20:51:23.188
2018/01/25 20:51:23.201 create-user
2018/01/25 20:51:23.215 create-user
2018/01/25 20:51:23.521 tag-user
incoming request…
saving user [theburningmonk] in the [user] table…
saved user [theburningmonk] in the [user] table
level
debug
debug
debug
debug tagging user [theburningmonk] with Azure Face API…
create-auth0-user

134. user
profile-images
POST /user
process-images
resize-images
image-tasks
Auth0
create-user
reformat-imagestag-user
Face API
Logs
timestamp component message
POST /user2018/01/25 20:51:23.188 incoming request…
level
debug
request-id
start-time
0ae4ba5d-dab1-4f9e-9de7-eace27ebfbc2
2018/01/25 20:51:23.188
method POST
create-auth0-user

135. user
profile-images
POST /user
process-images
resize-images
image-tasks
Auth0
create-user
reformat-imagestag-user
Face API
Logs
timestamp component message
2018/01/25 20:51:23.201 create-user
2018/01/25 20:51:23.215 create-user
2018/01/25 20:51:23.585
saving user [theburningmonk] in the [user] table…
saved user [theburningmonk] in the [user] table
level
debug
debug
debug uploading profile image…
create-user debug tagged user [theburningmonk] with Azure Face API…
create-user2018/01/25 20:51:23.587
create-auth0-user

136. user
profile-images
POST /user
process-images
resize-images
image-tasks
Auth0
create-user
reformat-imagestag-user
Face API
Logs
timestamp component message
2018/01/25 20:51:23.201 create-user
2018/01/25 20:51:23.215 create-user
2018/01/25 20:51:23.585
saving user [theburningmonk] in the [user] table…
saved user [theburningmonk] in the [user] table
level
debug
debug
debug uploading profile image…
create-user debug tagged user [theburningmonk] with Azure Face API…
create-user2018/01/25 20:51:23.587
click here to go to code create-auth0-user

138. Logs Input/Output
user
profile-images
POST /user
process-images
resize-images
image-tasks
Auth0
create-user
reformat-imagestag-user
Face API
input output
{
"body": "{ "username":"theburningmonk"}",
"resource": "/user",
"requestContext": {
"resourceId": "123456",
"apiId": “1234567890",
"resourcePath": "/user",
{
"statusCode": 200
}
create-auth0-user

139. Logs Input/Output
user
profile-images
POST /user
process-images
resize-images
image-tasks
Auth0
create-user
reformat-imagestag-user
Face API
input output
{ "Records": [
{ "Sns": {
"Type": "Notification",
"MessageId": "…",
"TopicArn": "…",
"Message": "…",
"Timestamp": "2018/01/25 20:51:24.215",
{
"error": null,
"result": "OK"
}
create-auth0-user

140. Logs Input/Output
user
profile-images
POST /user
process-images
resize-images
image-tasks
Auth0
create-user
reformat-imagestag-user
Face API
input error
{ "Records": [
{ "Sns": {
"Type": "Notification",
"MessageId": "…",
"TopicArn": "…",
"Message": "…",
"Timestamp": "2018/01/25 20:51:24.215",
[com.spaceape.dragon.handler.ReformatProfileImageHandle
r] Null reference exception
*java.lang.NullReferenceException: …
* at …
* at …
* at …
create-auth0-user

141. Logs Input/Output
user
profile-images
POST /user
process-images
resize-images
image-tasks
Auth0
create-user
create-auth0-user
reformat-imagestag-user
Face API
input error
{ "Records": [
{ "Sns": {
"Type": "Notification",
"MessageId": "…",
"TopicArn": "…",
"Message": "…",
"Timestamp": "2018/01/25 20:51:24.215",
[com.spaceape.dragon.handler.ReformatProfileImageHandle
r] Null reference exception
*java.lang.NullReferenceException: …
* at …
* at …
* at …
!

143. All
0 200 400 600 800
create-user
…user.insert_user
…user.upload_img
tag-user
create-auto0-user
process-images
resize-images
reformat-images!
837ms
406ms
66ms
114ms
122ms
82ms
240ms
157ms
35ms

144. All
0 200 400 600 800
create-user
…user.insert_user
…user.upload_img
tag-user
create-auto0-user
process-images
resize-images
reformat-images!
837ms
406ms
66ms
114ms
122ms
82ms
240ms
157ms
35ms

145. Input/Output
user
profile-images
POST /user
process-images
resize-images
image-tasks
Auth0
create-user
create-auth0-user
reformat-imagestag-user
Face API
Logs
!
All
0 200 400 600 800
create-user
…user.insert_user
…user.upload_img
tag-user
create-auto0-user
process-images
resize-images
reformat-images!
837ms
406ms
66ms
114ms
122ms
82ms
240ms
157ms
35ms

146. Input/Output
user
profile-images
POST /user
process-images
resize-images
image-tasks
Auth0
create-user
create-auth0-user
reformat-imagestag-user
Face API
Logs
!
All
0 200 400 600 800
create-user
…user.insert_user
…user.upload_img
tag-user
create-auto0-user
process-images
resize-images
reformat-images!
837ms
406ms
66ms
114ms
122ms
82ms
240ms
157ms
35ms

147. all your needs in one placeTRACING

149. mmm… it’s a graph

150. what if we can query it
like a graph?

152. http://amzn.to/2nk7uiW

153. ability to query based on the relationship
between observed components
(as well as the components themselves)

154. root cause analysis

155. the elevated error rate in service X was caused by
DynamoDB table throttling.“
”

156. payment was slow last
night around 10PM.
investigate.

157. time
95-percentile latency
service A
service B
10PM

158. time
95-percentile latency
service A
service B
10PM
causality? or correlation?

159. user-service
USESUSES
DEPENDS_ON
auth-serviceUSES
payment-service
DEPENDS_ON
“payment was slow last
night around 10PM”
user-table

160. user-service
USESUSES
DEPENDS_ON
auth-serviceUSES
DEPENDS_ON
payment-service
user-table
throttled exceptions!

161. user-table
user-stream
DEPENDS_ON
DEPENDS_ON USES
USES
USES
USES
USES
DEPENDS_ON
D
EPEN
D
S_O
N
DEPENDS_ON
PUBLISHES_TO
“what else is impacted by the throttled exceptions on user-table?”

162. user-table
user-stream
DEPENDS_ON
DEPENDS_ON USES
USES
USES
USES
USES
DEPENDS_ON
D
EPEN
D
S_O
N
DEPENDS_ON
PUBLISHES_TO
“what else is impacted by the throttled exceptions on user-table?”

163. wouldn’t that be nice?

165. MACHINE
LEARNING

166. use ML to auto-detect erroneous or
suspicious behaviours, or to suggest
possible improvements

168. !
Function [X] just performed an
unexpected write against
DynamoDB table [Y].
Should I…
ignore it from now on
shut it down!!

169. Observability Bot <bot@bestobservability.com>

170. Observability Bot <bot@bestobservability.com>
don’t bother me about this again

171. Observability Bot <bot@bestobservability.com>
auto-modify IAM role with DENY rule

172. Function [X]’s performance
has degraded since yesterday
- 99% latency has gone up by
47% from 100ms to 147ms.
!

173. !
Function [X] can run faster &
cheaper if you increase its
memory allocation.
Should I…
ignore it from now on
update setting

174. zzz… the future of… zzz …
serverless observability… zzz

176. Simon Wardley

177. Simon Wardley
context &
movement

178. However, I would argue that the health of the system no
longer matters. We've entered an era where what matters is
the health of each individual event, or each individual user's
experience, or each shopping cart's experience (or other high
cardinality dimensions). With distributed systems you don't
care about the health of the system, you care about the
health of the event or the slice.
”http://bit.ly/2E2QngU- Charity Majors
“

179. “one user action/vertical slice through the system”

180. movement
context
movement

181. The best way to predict the future
is to invent it.
- Alan Kay

182. The best way to invent
the future is to inception
someone else to do it.
- me