Serverless applications are the epitome of highly distributed, microservices applications. Execution happens everywhere - both inside and outside the serverless compute environment. For example, your functions could be triggered by an external service, then execute some code within AWS Lambda, then send a request over to a database, which *then* requires AWS Lambda to perform an update in a second data store. You might be able to predict and design for certain troublesome issues but there are many, many more that you probably will not be able to easily plan for. How do you build a resilient system under these highly distributed circumstances? The answer is chaos engineering. Join us as we walk through: The unique challenges of building a highly resilient serverless app Why you need to design for problems you cannot predict and cannot easily test for How you can use chaos engineering to build a resilient serverless application How you can take advantage of out of the box and third-party observability solutions to measure the impact of chaos experiments.

1. @emrahsamdan
Applying Chaos Engineering to build resilient
serverless applications
Emrah Şamdan
(@emrahsamdan)
11/6/2019

2. Who am I?
● Developer for 6+ years
● Product guy for 2 years
● VP of Product for Thundra
● Organizing committee
● Father of a chaos monkey

3. @emrahsamdan
Why chaos engineering?
Unit Tests
● My function is running properly
and meets the expectations.
Integration Tests
● My system is running properly
and meets the expectations.
UI/UX Tests
● It is like a charm!

4. @emrahsamdan
Why chaos engineering?
Unit Tests
● My function is running properly
and meets the expectations.
Integration Tests
● My system is running properly
and meets the expectations.
UI/UX Tests
● It is like a charm!

5. @emrahsamdan

6. @emrahsamdan

7. @emrahsamdan
Your third party API slows down so badly..

8. @emrahsamdan
Some part of your system becomes unreachable.

9. @emrahsamdan
Your cache/DB is down so you can’t load your data.

10. @emrahsamdan
Chaos Engineering is the discipline of experimenting on a system
in order to build confidence in the system’s capability
to withstand turbulent conditions in production.
http://principlesofchaos.org/

11. @emrahsamdan
Breaking things on purpose in production

12. @emrahsamdan
Breaking things on purpose in production
To make them more resilient

13. @emrahsamdan
Breaking things on purpose in production
To make them more resilient
Well, maybe instaging?

14. @emrahsamdan
Chaos Engineering is
Vaccine to software

15. @emrahsamdan
Chaos Engineering is
Vaccine to software For resiliency

16. @emrahsamdan
Chaos Engineering is
Vaccine to software For resiliency To prevent outages

17. @emrahsamdan
Chaos Engineering is
Vaccine to software For resiliency To prevent outages To define steady state

18. @emrahsamdan
Chaos Engineering is not
For breaking down

19. @emrahsamdan
Chaos Engineering is not
For breaking down For bad surprises

20. @emrahsamdan
Chaos Engineering is not
For breaking down For bad surprises For blaming

21. @emrahsamdan
Chaos Engineering is not
For breaking down For bad surprises For blaming For causing outages

22. @emrahsamdan

23. @emrahsamdan
History of chaos engineering?
2010 2011 2014 2019

24. @emrahsamdan
Companies applying Chaos Engineering

25. @emrahsamdan
Ups and downs of the
system.
If I take this server down,
maybe everything will still
run smooth. Maybe?
Let me attack on my system!
Cute! Let me break
something else.
Oh! I should fix this before it
actually happens and then
break something else.

26. @emrahsamdan
Chaos experiments will(should) never end!

27. @emrahsamdan
Don’t break on purpose!
● Start experimenting with the first row, the
leftmost cell: Known-knowns.
● Blast radius: The effect will make the
smallest effect.
● Put a stop button somewhere!
● Plan how you learn.
● You don’t need to do it on production for
the first time.
● The most important Let the other people
know! Surprising chaos is not funny. No, at
all!

28. @emrahsamdan
Chaos examples
● Your system keeps records on the DB.
● DB is returning too slow for 1% of your customers.
Hypothesis: The system won’t experience an outage when DB is hardly
accessible.

29. @emrahsamdan
Chaos examples
● Your system keeps records on the DB.
● DB is returning too slow for 1% of your customers.
Hypothesis: The system won’t experience an outage when DB is hardly
accessible.
Result: People experiences timeouts while waiting for results.

30. @emrahsamdan

31. @emrahsamdan
You never fail!

32. @emrahsamdan
Chaos when everything is more granular.
SERVERLESS

33. @emrahsamdan
More Granular Functions

34. @emrahsamdan
More Granular Functions

35. @emrahsamdan
More Granular Functions

36. @emrahsamdan
Every service has its own failure mode
Lots of managed intermediate service which has its own bad-day
characteristics.
Different throttling, different retry mechanisms for different services.

37. @emrahsamdan
Every function has its own configuration
● Timeouts
● IAM Roles

38. @emrahsamdan

39. @emrahsamdan
Common weaknesses in serverless
● Nested functions with improper timeouts

40. @emrahsamdan
Common weaknesses in serverless
● Unhandled errors from upstream services

41. @emrahsamdan
Common weaknesses in serverless
● Failures in resources

42. @emrahsamdan
Chaos experiments in serverless
● Inject latency to downstream services
● Inject failure to resources

43. @emrahsamdan
Injecting latency
● Don’t attack your system.
● You don’t need to do on prod
first.
● There is no point to inject
latency to async calls.
Hypothesis: Entry point Lambda will
degrade gracefully when the
downstream Lambda times out or turns
really late.

44. @emrahsamdan
Where else to inject?
Inject latency to resources, too.

45. @emrahsamdan
How to inject latency

46. @emrahsamdan
How to inject latency with Thundra

47. @emrahsamdan
Injecting Error
● Connection errors with third party services
● Cache down
● AWS Resource is unreachable

48. @emrahsamdan
What if we lose the connection to Redis?

49. @emrahsamdan
Let’s inject error to Redis with Thundra

50. @emrahsamdan
Common fixes
● Exponential backoff
● Properly tunes timeouts
● Circuit breakers
● Use async communication when possible

51. @emrahsamdan
Don’t forget! Aim is
● Not to break but to improve
● Not to blame people but to give them room to fix
● Not to surprise your colleagues but to make your system resilient

52. @emrahsamdan
Thank you!