In this talk, we will discuss some tips for alerting around your serverless application, and different approaches to troubleshooting issues in your serverless application: using first-party tools from AWS; using custom-built solutions; or using a serverless monitoring solution.

1. Serverless Application
Troubleshooting

2. I watch a lot of TV shows…

3. protagonist is shot

4. 3 hours earlier… protagonist is shot

5. 3 hours earlier… protagonist is shot

6. 3 hours earlier… protagonist emerges
victoriously
protagonist is shot

7. happened

8. happened user impact

9. happened system repaireduser impact

10. happened system repaireduser impact
goal: to fail without users noticing

11. happened system repaireduser impact
reduce MTTR

12. Yan Cui
http://theburningmonk.com
@theburningmonk
Developer Advocate @
Independent Consultant
AWS user since 2009
since 2018
yan@lumigo.io

13. What do you mean
by ‘serverless’?

14. “Serverless”

16. Gojko Adzic
It is serverless the same way
WiFi is wireless.
http://bit.ly/2yQgwwb

17. Serverless means…
don’t pay for it if no-one uses it
don’t need to worry about scaling
don’t need to provision and manage servers

18. in other words, it’s a lot like taking a cab

19. Ownership
Fuel
Navigate
To get there!
Focus on
getting there!

20. HW Ownership
OS
Runtime & Scale
Code
Focus on
getting there!
Physical
Servers
Virtual
Machines
Containers Serverless

21. Nano Services Self Managed Cost Paradigm
ChangeAsync
Dynamic agile env

22. happened system repaireduser impact
reduce MTTR

23. Identify & Resolve
Issues
Understanding
costs
Visibility

24. Identify & Resolve
Issues
Understanding
costs
Visibility

25. happened system repaireduser impact
MTTDiscovery

27. “What alerts should I have?”

28. It depends on what you’re building…

29. But, this is a good starting point

30. Lambda
error rate %
throttle count
DLR error count
iterator age
regional concurrency

31. Lambda
error rate %
throttle count
DLR error count
iterator age
regional concurrency
API Gateway
p90/95/99 latency
success rate %
4xx rate %
5xx rate %

32. API Gateway
p90/95/99 latency
success rate %
4xx rate %
5xx rate %
SQS
message age
Lambda
error rate %
throttle count
DLR error count
iterator age
regional concurrency

33. API Gateway
p90/95/99 latency
success rate %
4xx rate %
5xx rate %
SQS
message age
Step Functions
failed count
throttle count
timed out count
Lambda
error rate %
throttle count
DLR error count
iterator age
regional concurrency

34. SQS
message age
Step Functions
failed count
throttle count
timed out count
API Gateway
p90/95/99 latency
success rate %
4xx rate %
5xx rate %
Lambda
error rate %
throttle count
DLR error count
iterator age
regional concurrency

35. “Can’t you codify these?”

37. Identify & Resolve
Issues
Understanding
costs
Visibility

38. happened system repaireduser impact
finding root cause

39. option 1: CloudWatch & friends

41. https://lumigo.io/blog/getting-the-most-out-of-cloudwatch-logs/

42. Pros
Out of the box
No overhead
Comparatively cheap
AWS support

43. Pros
Out of the box
No overhead
Comparatively cheap
AWS support
Cons
Complicated

44. https://lumigo.io/blog/serverless-applications-automate-chores-cloudwatch-logs/

45. Pros
Out of the box
No overhead
Comparatively cheap
AWS support
Cons
Complicated
Hard to query*
* Insights improved things drastically, but still a gap to ELK

48. https://lumigo.io/blog/how-to-monitor-lambda-with-cloudwatch-metrics/

49. Pros
Out of the box
Source of truth
No overhead*
Comparatively cheap
AWS support
* unless you record custom metrics synchronously

50. Pros
Out of the box
Source of truth
No overhead*
Comparatively cheap
AWS support
* unless you record custom metrics synchronously
** can compensate with custom metrics/metric filters, etc.
Cons
Missing metrics**
Lambda percentile
latencies don’t work
Only granular to 1 min
No query language

52. Pros
Out of the box
SDK
No overhead
Comparatively cheap
AWS support

53. Pros
Out of the box
SDK
No overhead
Comparatively cheap
AWS support
Cons
Poor async support

55. Pros
Out of the box
SDK
No overhead
Comparatively cheap
AWS support
Cons
Poor async support
No auto-
instrumentation
Bad DX (for node.js)
Poor documentation

56. option 2: custom built solutions

57. https://github.com/getndazn/dazn-lambda-powertools

59. Structured Logging

60. Structured Logging
Sampling

61. Structured Logging
Sampling
Correlation IDs

62. Structured Logging
Sampling
Correlation IDs
Auto “instrumentation”

63. Structured Logging
Sampling
Correlation IDs
Auto “instrumentation”
Support async events

64. enrich the usefulness of your logs

65. https://theburningmonk.com/2017/08/centralised-logging-for-aws-lambda/

66. https://theburningmonk.com/2018/07/centralised-logging-for-aws-lambda-revised-2018/

68. Pros
Tailor fit
Free!

69. Pros
Tailor fit
Free!
Cons
Very high-touch
Not all services are
supported equally
Tailor fit (for someone
else…)

70. option 3: serverless monitoring solutions

74. Pros
SAAS
Serverless focus
More than just tracing
Very low touch
Cons
Yet another 3rd party
More than just tracing

75. Takeaways
Serverless is a game-changer
Serverless has challenges
Options for troubleshooting serverless applications

76. https://info.lumigo.io/serverless-consulting
Start off on the right foot

78. @theburningmonk
theburningmonk.com
github.com/theburningmonk
yan@lumigo.io