This document discusses chaos engineering and using chaos experiments to test systems reliability. It introduces chaos engineering as a way to acquire knowledge about how systems behave under stressful conditions in order to make better decisions. Chaos experiments follow a protocol of defining a baseline, impacting the system in a controlled way ("what if..." experiments), and observing if the impact deviates from the baseline. The Chaos Toolkit is presented as an open source tool that automates running chaos experiments across different platforms and services. It has drivers for common platforms and tools to inject faults and monitor system behavior. Chaos engineering helps build more reliable, secure and performant systems.

1. Systems Reliability with
Chaos Engineering
Sylvain Hellegouarch
ChaosIQ CTO
@chaosiqio

2. Chaos Engineering
is Knowledge
Better decision making

3. Actionable “what if…?”
experiments
HOW DO WE ACQUIRE DATA?

4. System properties
which impact your
users and business
Safety, availability, security, SLI...
WHAT IF..?

5. Number of purchases
per second remains
the same
Performability, Reliability
WHAT IF WE INCREASE LATENCY BY 500ms?

6. An alert should be
triggered before users
get impacted
Security, Safety
WHAT IF AN INTERNAL TLS CERT EXPIRES?

7. User must keep access
to the service
Availability, Reliability
WHAT IF WE LOSE A COMPUTING NODE?

8. Experiment is a
protocol
Phase 1: Define a baseline
PROVE OR DISPROVE YOUR “WHAT IF…”

9. Experiment is a
protocol
Phase 2: Impact the system(what if…?)
PROVE OR DISPROVE YOUR “WHAT IF…”

10. Experiment is a
protocol
Phase 3 :Has the baseline deviated?
PROVE OR DISPROVE YOUR “WHAT IF…”

11. Add latency
to one of the links
Baseline: Services should be healthy and we
have X purchases/second
EXPERIMENT

12. Add latency
to one of the links
Deviation: Did we see timeouts errors?
Did we notice a loss in business?
EXPERIMENT

13. Load an expired
certificate
Baseline: There should be no active alerts
for that error
EXPERIMENT

14. Load an expired
certificate
Deviation: Did we get an alert for that error?
EXPERIMENT

15. Terminate a node
at random
Baseline: We can access the service
EXPERIMENT

16. Terminate a node
at random
Deviation: Can we access the service?
EXPERIMENT

17. Automated execution
Use dedicated tooling
ACTIONABLE

18. Automated execution
Be part of the engineering routine
Like linting, testing, CI/CD, DevOps...
ACTIONABLE

19. Learn new knowledge
Make better decisions
SURFACE THE RESULTS

20. How do we proceed?

21. Chaos as Code

22. Open Source
Apache v2 License

23. Active Community
Civil and Friendly

25. Small image
This is a caption.
A shadow can make the image stand out more. Check the
settings on this one as a reference.

26. Automates the Chaos
Engineering protocol
With extensibility in mind

27. Rich Set of Drivers
Native API and Fault Injection tools

28. Rich Set of Drivers
AWS, Azure, Cloud Foundry, Kubernetes, GCE, Gremlin, Prometheus,
Humio, ToxiProxy, Spring Chaos Monkey...

29. Supports Observability
Monitoring, Tracing, Logging, Alerting, Slack

30. Reporting
Transforms collected data into knowledge

31. What does it look like?

35. Metadata meant to communicate
the experiment’s purpose.
Communication is key

36. Experiments are dynamic
Configuration and secrets filled at runtime

37. Declare your baseline
Probes validate the system
is in a steady-state

38. Impact your system
Actions should put the system in degrated
mode to validate our hypothesis

39. Impact your system
Incidentally, the sole mandatory declaration
of an experiment

40. Contributions
How does this particular experiment benefit
our knowledge of our system’s properties?

41. Distribution of areas impacted by these contributions
availability
durability
observability
recoverability
reliability
safety
security
Kubernetes
Spring
Database
Service

42. Error Budgets
Contribution +
Reporting

43. There is more!

44. Rollbacks

45. Controls

46. Chaos Engineering
Control Plane
Scheduling and Experiments Management

47. Chaos Engineering
Platform
Enable Your Chaos Engineering Capability

48. Drivers

49. Application Level
Spring Chaos Monkey

50. Network Level
ToxiProxy

51. Platform Level
Kubernetes, Cloud Foundry

52. Infrastructure Level
AWS, Cloud Foundry, Google Cloud, Azure

54. Plugins
Reporting, Slack, ChaosHub, Tracing

57. More in the future :)https://github.com/chaostoolkit/chaostoolkit/issues/74

58. Demo!

59. Copyright Marvel

60. Super Power App!
HTTPUsers Superpower SuperpowerHTTP

61. What does it do?
•The first time users access it, a superpower is attached to
them
•That superpower must remain the same all along (we can’t
have you change superpowers, can we?)
•It must be return you a power in very rapidly

62. one of our services goes down?
WHAT IF..?

63. an internal dependency
goes down?
WHAT IF..?

64. new version fails?
WHAT IF..?

65. a node in the platform
goes down?
WHAT IF..?

66. a link between nodes
gets slower?
WHAT IF..?

67. we cannot reach
an external third-party?
WHAT IF..?

68. Experiments
•Terminate one instance of the powersource service
•Terminate both instances
•Inject heavy load into the server and measure the impact
perceived by our super awesome users

69. https://da67470c.ngrok.io/
Please visit:

70. Thank you!
Visit https://chaosiq.io for more
informations