This document discusses how DevSecOps and chaos engineering can be used to test systems and build confidence in their ability to withstand turbulent conditions. It provides an overview of how a large healthcare company faces challenges due to its size, complexity, and diverse technology portfolio. The document advocates using chaos engineering experiments to gain objective understanding of security, validate security incident response plans, discover new insights about security tools and processes, and build a learning culture around security. It acknowledges that systems are becoming more unpredictable and difficult for humans to understand, and that chaos engineering can help determine how security defenses actually work.

1. DevSecOps & Chaos Engineering
“Knowing the Unknown”
@aaronrinehart

2. Aaron Rinehart
Chief Security Architect
UnitedHealth Group
Contact Info
Rinehart.Aaron@gmail.com
@aaronrinehart
2

3. IN This Session We will Cover

4. • Fortune 6 Company
• 380+ Companies & Growing
• 28,000+ Developers
• 8,000+ Applications
• Highly Regulated
• Diverse Technology Portfolio
• Security Testing: Mostly
Human Driven
• Cloud Journey: Mixed
The Challenge: We are Large & Complex
• Largest Private HealthCare
Company in World
• 1000+ Security
Professionals
• Multinational Business
• Some DevOps
• Waterfall, Agile, & Others

5. 2018 Causes of Data Breaches

6. Security
Incidents are
Subjective in
Nature

7. We don't know
Where? Why? Who?
What?How?
very much

8. How do we find
out typically?

9. SECURITY INCIDENTS
ARE NOT
DETECTIVE MEASURES

10. IS NOT A STRATEGY

11. Teams spend too
much time
reacting to
outages instead
of building more
resilient systems.

12. Build Confidence
in
What Actually
Works

13. “Chaos Engineering is the discipline of
experimenting on a distributed system
in order to build confidence in the
system’s ability to withstand turbulent
conditions”

14. Netflix Chaos ToolsChaos Monkey

15. Who is doing Chaos?

16. Don’t just test
….Experiment

17. Testing
vs.
Experimentation

18. Use
Chaos Engineering
to derive
Objective
Understanding about
Security

19. Validate Security Incident Runbooks
Discover new insights about Security
Toolchains
Build a Learning Culture around Security
Drive Security Transparency
Safely explore failures within system defenses
Drive out redundancy
Discover System Observability Gaps
Improve Quality of System Events

20. Focus is on the immediate remediation
of the problem not the deeper
understanding of how the system is
behaving
Fail Forward.

21. Failure Happens.

22. Build a
Learning Culture
around
Failure

23. Humans &
Systems
need
failure to
learn &
Grow

24. Our Systems are becoming:
- Unpredictable
- Non-linear
- Immutable
- Complex
- Highly Distributed
- Rapidly Iterative
Changes
- Difficult for Humans

25. Security is still
- Mostly Monolithic
- State Dependant
- Stateful in Stateless World
- Preventative Design
- Complex
- Poorly Aligned

26. How does My Security
Really Work?

27. Am I Sure it Works
that way?

28. How would I know?

29. 29
An Open Source Tool

30. • ChatOps Integration
• Configuration-as-Code
• Example Code & Open
Framework
ChaoSlingr Product Features
• Serverless App in AWS
• 100% Native AWS
• Configurable Operational
Mode & Frequency
• Opt-In | Opt-Out Model

31. Questions