Everything fails all the time! A quote repeated by many everyday. How does it feel when things fail in production? How do you recover from such situations? How can you make sure they don’t repeat? All these discussed with real production incidents and the measures taken to mitigate such failures. We will also look at few of the most common failure possibilities in a serverless ecosystem. Remember, when everything fails all the time, you must learn something everyday to be operational all the time!

1. From Oops… to Ops
Sloppy Little Serverless Stories
Sheen Brisals
The LEGO Group
sheenbrisals

2. Oops!
A word that is used to show
the recognition of a mistake
Ops
Operations

4. LEGO.com was
switched to
Serverless
on AWS Cloud
July 10, 2019

5. 200+ Lambda functions
40+ microservices
40+ API endpoints
25+ DynamoDB tables
30+ S3 buckets
20+ SNS topics
60+ SQS queues + DLQs
200+ SSM parameters
PROD - Services Stats

6. Oops! Moments…

7. No one starts perfect with Serverless.
That’s fine, but strive to be better at
every iteration, and that is important.

8. Oops!
1
When friendly
Firehose turned
foe…

9. Kinesis
Data Firehose
API
Gateway
S3
bucket
Event
producer
Fan-out
function
Click-stream event ingestion
Buffer size Buffer interval

10. Oops! Ops
Buffer size: 3 MB
Buffer interval: 1 min
Peak season lambda
failure
Buffer size: 1 MB
Buffer interval: 1 min
Perfect with a tuned
lambda

11. Oops!
2
When SSM
Parameter Store
packed a punch…

12. 200+ Lambda functions
40+ microservices
40+ API endpoints
25+ DynamoDB tables
30+ S3 buckets
20+ SNS topics
60+ SQS queues + DLQs
200+ SSM parameters
PROD - Services Stats

13. 200+ Lambda functions
40+ microservices
40+ API endpoints
25+ DynamoDB tables
30+ S3 buckets
20+ SNS topics
60+ SQS queues + DLQs
200+ SSM parameters
PROD - Services Stats

14. 200+ Lambda functions
200+ SSM parameters
PROD - Services Stats
Reads Writes

15. Oops! Ops
Default throughput
40 TPS
Rate limiting

16. Oops! Ops
Default throughput
40 TPS
Rate limiting
Advanced throughput
1000 TPS
Self change via console

17. Oops!
3
When too much
became
too little…

18. CloudWatch
Logs
Balancing concurrency
Concurrency 25
Log
Splitter
Elasticsearch
Monitoring

19. Oops! Ops
Failing at wrong time
Missing crucial logs
Inadequate testing
Incorrect dashboards
Better dev
process
Better
monitoring

20. Oops!
4
When going
higher gone
through the roof…

21. CloudWatch
Event
Heavy lifting
function
Art of coding – Copy & Paste
Trigger
rule
StepFunction
2 GB RAM
5 mins run
2 x daily
Frontend Status check
API
Request
handler
Status
store
2 GB RAM
100 ms run
1000s x daily

22. Oops! Ops
Memory: 2 GB
Invocations: 1 per sec
Invocations: 2.5mil /mo
Cost~ $9.00 / month
Duration: 100 ms
Memory: 256 MB
Invocations: 1 per sec
Invocations: 2.5mil /mo
Cost~ $1.50 / month
Duration: 100 ms

23. Oops!
5
When a key
moment turned
chaotic…

24. SaaS
API
Feeds
Feeds
store
DLQ
Data Pipeline
Internal
Feeds API
Feeds
Internal
App
{API Key}
The API re-deployment somehow
changed the API Key!
No one knew. No one noticed!

25. Oops! Ops
Silent breakage
Bad customer exp.
Unhappy business
Chaotic dev process
Better dev
process
Better
prompts
Better
monitoring

26. Oops! Oopsibilities…

27. Serverless requires a new way of thinking, new way of
working, and new way of running applications.
That means, we need to change our way of thinking, our
way of working, and our way of running applications.

28. Oopsibility!
1
High load
hiccup

29. Oopsibility!
1
High load
hiccup

30. Oopsibility!
2
Dev to Prod
debacle
• Keys & secrets
• Domains
• Resource Configurations
• Leaked privacy

31. Oopsibility!
3
Admin
Access
Abuse

32. Oopsibility!
3
Admin
Access
Abuse

33. Oopsibility!
4
Ignorance
is
blip

34. Oopsibility!
5
Third-party
trauma

35. Oopsibility!
5
Third-party
trauma
SaaS

36. From Oops of Sorrows to
Operational Success…
• Know the service limits
• Be Well-Architected
• See through the Serverless Lens
• Alarm – Alert – Act
• Monitor monitor monitor

37. Thank
you!
Go
Build
Serverless
sheenbrisals