With serverless applications, execution can happen everywhere. It’s hard to predict and design for all troublesome issues. Chaos engineering can help you build highly resilient systems. Join us to learn how and why this approach is especially valuable when building serverless applications.

1. Applying Chaos Engineering to build resilient
serverless applications
Emrah Şamdan
(@emrahsamdan)
4/25/2019

2. Who am I?
● Developer for 6+ years
● Product guy for 2 years
● VP of Product for Thundra
● Organizing committee
● Serverlessdays İstanbul
On October 11st!

3. Agenda
● What’s chaos engineering?
● Why chaos testing on serverless?
● Best practices on chaos testing for serverless
● How to apply chaos testing on AWS Lambda
● How to apply silence in a world of chaos

4. Why chaos engineering?
Unit Tests
● My function is running properly
and meets the expectations.
Integration Tests
● My system is running properly
and meets the expectations.
UI/UX Tests
● It is like a charm!

5. Why chaos engineering?
Unit Tests
● My function is running properly
and meets the expectations.
Integration Tests
● My system is running properly
and meets the expectations.
UI/UX Tests
● It is like a charm!

8. Your third party API slows down so badly..

9. Some part of your system becomes unreachable.

10. Your cache/DB is down so you can’t load your data.

11. Chaos Engineering is the discipline of experimenting on a system
in order to build confidence in the system’s capability
to withstand turbulent conditions in production.
http://principlesofchaos.org/

12. Chaos Engineering is
● Like injecting vaccine to your system to make it more
immune
● To improve your system’s resilience by uncovering
weaknesses.
● Identifying failures before they become outages.
● To understand the steady state of your system and
challenge it.

13. Chaos Engineering is not
● Breaking down production for purpose.
● For blaming a group of people.
● Surprising your colleagues with partial outages.
● Taking down all the system at the same time.

15. History of chaos engineering?
2010 2011 2014 2019

16. Companies applying Chaos Engineering

17. States of chaos engineering
● Define steady state
● Hypothesis on steady state of the system with the designed failure
● Run your experiment
○ Define blast radius
○ Define halting condition
○ Have a rollback plan!
● Verify & Learn
○ If your system breaks you understood an issue before it causes an outage. Go fix it!
○ If it is resilient, congrats! Now, inject some other failure!

18. Don’t break on purpose!
● Start experimenting with the first row, the
leftmost cell: Known-knowns.
● Blast radius: The effect will make the
smallest effect.
● Put a stop button somewhere!
● Plan how you learn.
● You don’t need to do it on production for
the first time.
● The most important Let the other people
know! Surprising chaos is not funny. No, at
all!

19. Chaos examples
● Your system keeps records on the DB.
● DB is returning too slow for 1% of your customers.
Hypothesis: The system won’t experience an outage when DB is hardly
accessible.

20. Chaos examples
● Your system keeps records on the DB.
● DB is returning too slow for 1% of your customers.
Hypothesis: The system won’t experience an outage when DB is hardly
accessible.
Result: People experiences timeouts while waiting for results.

22. You never fail!

23. Chaos when everything is more granular.
SERVERLESS

24. More Granular Functions

25. More Granular Functions

26. More Granular Functions

27. Every service has its own failure mode
Lots of managed intermediate service which has its own bad-day
characteristics.
Different throttling, different retry mechanisms for different services.

28. Every function has its own configuration
● Timeouts
● IAM Roles

30. What would you do when your region is down?

32. Common weaknesses in serverless
● Nested functions with improper timeouts

33. Common weaknesses in serverless
● Unhandled errors from upstream services

34. Common weaknesses in serverless
● Failures in resources

35. Chaos experiments in serverless
● Inject latency to downstream services
● Inject failure to resources

36. Injecting latency
● Don’t attack your system.
● You don’t need to do on prod
first.
● There is no point to inject
latency to async calls.
Hypothesis: Entry point Lambda will
degrade gracefully when the
downstream Lambda times out or turns
really late.

37. Where else to inject?
Inject latency to resources, too.

38. How to inject latency

39. Injecting Latency to resources by Yan Cui

40. How to inject latency with Thundra

41. Injecting Error
● Connection errors with third party services
● Cache down
● AWS Resource is unreachable

42. What if we lose the connection to Redis?

43. Let’s inject error to Redis with Thundra

44. Common fixes
● Exponential backoff
● Properly tunes timeouts
● Circuit breakers
● Use async communication when possible

45. Don’t forget! Aim is
● Not to break but to improve
● Not to blame people but to give them room to fix
● Not to surprise your colleagues but to make your system resilient

46. Thank you !