The journey towards confidence in your system and team through the "Try, Learn, Adapt" flow of Chaos Engineering.

1. Build Confidence in your
System with Chaos
Engineering
Try, Learn, Adapt
2017 - Sylvain Hellegouarch - http://chaosiq.io CHAOSIQ

2. Where we started...

3. To build this

4. What we tend to actually build

5. The environment matters... not?

6. Well, yes it does!

7. But, I’m prepared!

8. Even prepared, things may go wrong...
Kudos for
the report!

9. Everything changes and nothing stands still
Heraclitus - 402BC

10. Non-functional is a force towards change
It’s fine, we
have
microservices

11. Adapt

12. Things are fine… so far

13. As soon as you are launched you are at
the mercy of your environment

14. Environment is full of unknowns

15. "It's just one of these cases where Mars is going to give us
a new deal, and we're going to have to play the cards we
get, not the ones we want”
Jim Erickson / Project Manager at Nasa for Mars Rovers missions

16. Strategies?

17. Robustness
Resist to change

18. Redundancy
Duplicate Components

19. Adaptation
Dynamic Response to Environment

20. One might rephrase “calculation and
correction of error”

21. One might rephrase “calculation and
correction of error” as “recognition of
and response to difference”.Jeff Sussna / Designing Delivery: Rethinking IT in the Digital Service Economy

22. Your System is dynamic and fluid
Recognition

23. Environment is not stable
Recognition

24. Users do not care
Recognition

25. Accept you do not control everything
Recognition

26. Double-learning loop
Response

27. Be creative when you explore but seek
evidence
Response

28. Learn

29. What to learn?

30. Look for deviation from normal

31. Context sensitive

32. Strategies to learn

33. It’s not about breaking stuff you fool!

34. Open Ended Questions
Chaos Engineering

35. Probe your system for data
Chaos Engineering

36. Chaos Engineering is about trying
controlled change to observe system
availability deviation

37. Game Days

38. Planned experiments of realistic events

39. Collective Effort
Shared Understanding

40. Communication is key
No surprises

41. Generate a Playbook
Automation helps confidence

42. Observability is Key

43. Collect Metrics

44. Never enough logging

45. Be careful when interpreting data

46. Be Creative

47. You can't always get what you want
But if you try sometimes you might find
The Rolling Stones

48. Try

49. chaostoolkit.org
Chaos as a Code

50. What normal looks like? Your steady state
{
"probes": {
"steady": {
"title": "All services must be healthy before we begin",
"layer": "application",
"type": "python",
"module": "chaosk8s.probes",
"func": "all_microservices_healthy"
}
}
}

51. Add sources of information with probes
"probes": {
"close": {
"title": "Fetch the CPU usage for our service",
"layer": "application",
"type": "python",
"module": "chaosprometheus.probes",
"func": "query",
"arguments": {
"query": "process_cpu_seconds_total{job='websvc'}",
"when": "2 minutes ago"
}
}
}

52. Set the condition for change in normality
"action": {
"title": "Let's max out the CPU of a node",
"layer": "application",
"type": "python",
"module": "chaosgremlin.actions",
"func": "attack",
"secrets": "gremlin",
"arguments": {
"command": {
"type": "cpu"
},
"target": {
"type": "Random"
}
}
}

53. Before learning
$ chaos run experiment.json
[2017-10-06 17:37:33 INFO] Running experiment: System is resilient to provider's failures
[2017-10-06 17:37:33 INFO] Observing steady state: All services must be healthy before we begin
[2017-10-06 17:37:33 INFO] Steady State succeeded
[2017-10-06 17:37:33 INFO] Observing steady state: Before we kill it, our microservice should be alive
[2017-10-06 17:37:33 INFO] Steady State succeeded
[2017-10-06 17:37:33 INFO] Observing action: Let's stop our provider
[2017-10-06 17:37:33 INFO] Action succeeded
[2017-10-06 17:37:33 INFO] Observing close state: All services must be healthy before we begin
[2017-10-06 17:37:33 INFO] Close State succeeded
[2017-10-06 17:37:33 INFO] Observing steady state: Consumer should respond as if nothing
[2017-10-06 17:37:44 ERROR] Steady State failed: {"timestamp":1507304264100,"status":500,"error":"Internal
Server Error","exception":"feign.RetryableException","message":"connect timed out executing GET http://my-
provider-service:8080/","path":"/invokeConsumedService"}
[2017-10-06 17:37:44 INFO] Experiment is now complete

54. Respond to the non-functional force of
change
Do not merely correct the error

55. Adaptation
$ chaos run experiment.json
[2017-10-06 17:40:25 INFO] Running experiment: System is resilient to provider's failures
[2017-10-06 17:40:25 INFO] Observing steady state: All services must be healthy before we begin
[2017-10-06 17:40:25 INFO] Steady State succeeded
[2017-10-06 17:40:25 INFO] Observing steady state: Before we kill it, our microservice should be alive
[2017-10-06 17:40:26 INFO] Steady State succeeded
[2017-10-06 17:40:26 INFO] Observing action: Let's stop our provider
[2017-10-06 17:40:26 INFO] Action succeeded
[2017-10-06 17:40:26 INFO] Observing close state: All services must be healthy before we begin
[2017-10-06 17:40:26 INFO] Close State succeeded
[2017-10-06 17:40:26 INFO] Observing steady state: Consumer should respond as if nothing
[2017-10-06 17:40:30 INFO] Steady State succeeded
[2017-10-06 17:40:30 INFO] Experiment is now complete

56. CHAOSIQ
Chaos Engineering for Cloud Native
SaaS & On-Premises

57. Thank You

58. Sylvain Hellegouarch
CTO ChaosIQ
@lawouach / sylvain@chaosiq.io