This document summarizes a presentation on chaos engineering and security chaos engineering. It discusses how systems have become too complex for humans to fully understand and that failures are the normal condition. Chaos engineering experiments intentionally introduce failures to build confidence in a system's resilience. Security chaos engineering uses the same principles to continuously validate security controls and reduce uncertainty. The document provides examples of chaos experiments and introduces ChaoSlingr, an open source tool for automating security chaos experiments.

1. @aaronrinehart @verica_io #chaosengineering
Security
Chaos Engineering

2. @aaronrinehart @verica_io #chaosengineering
● Combating Complexity
in Software
● Chaos Engineering
● Resilience Engineering
& Security
● Security Chaos Engineering
Areas Covered

3. 3
Aaron Rinehart, CTO, Founder
● Former Chief Security Architect @UnitedHealth
● Former DoD, NASA Safety & Reliability
Engineering
● Frequent speaker and author on Chaos
Engineering & Security
● O’Reilly Author: Chaos Engineering, Security
Chaos Engineering Books
● Pioneer behind Security Chaos Engineering
● Led ChaoSlingr team at UnitedHealth
@aaronrinehart @verica_io #chaosengineering

4. Incidents,Outages, &
Breaches are
Costly

5. An Obvious
Problem

6. Why do they
seem to be
happening
more often?

7. Our systems have evolved beyond human
ability to mentally model their behavior.
7

8. 8
everyone else
Our systems have evolved beyond human
ability to mentally model their behavior.

10. Circuit Breaker Patterns
10
Continuous
Delivery
Distributed
Systems
Blue/Green
Deployments
Cloud
ComputingService Mesh
Containers
Immutable
Infrastructure
Infracode
Continuous
Integration
Microservice
Architectures
API Auto Canaries
CI/CD
DevOps
Automation Pipelines
Complex?

11. Mostly
Monolithic
Requires
Domain
Knowledge
Prevention
focused Poorly Aligned
Defense in
Depth
Stateful in
nature
DevSecOps not
widely adopted
Security?
Expert
Systems
Adversary Focused

12. Simplify?

13. Software
has
officially
taken over

14. Software Only Increases in
Complexity

15. What does this
have to do with
my systems?

16. Question How well do you
really understand
your own systems?

17. Systems
Engineering is
Messy
In Reality…….

18. In the
beginning...we
think it looks
like

19. After a few
months….
Hard Coded Passwords
Identity Conflicts
Lead Software
Engineering finds a new
job at Google
New Security Tool
Refactor Pricing
300 Microservices Δ-> 850 Microservices
Cloud Provider API
Outage
WAF Outage -> DisabledScalability Issues
Network is Unreliable
Autoscaling Keeps
Breaking
Large CustomerDelayed Features
DNS Resolution
ErrorsExpired Certificate
Regulatory
Audit
Rolling Sev1 Outage
on Portal
Code Freeze

20. Years?….
Hard Coded Passwords
Identity Conflicts
Lead Software Engineering finds
a new job at Google
New Security Tool
Refactor Pricing
300 Microservices Δ-> 4000 Microservices
Cloud Provider API Outage
Firewall Outage -> Disabled
Scalability Issues
Network is Unreliable
Autoscaling Keeps
Breaking
Large Customer Outage
Delayed Features
DNS Resolution ErrorsExpired Certificate
Regulatory Audit
Rolling Sev1 Outages on
Portal
Code Freeze
Hard Coded Passwords
Identity Conflicts
Lead Software Engineering finds
a new job at Google
New Security Tool
Refactor Pricing
300 Microservices Δ-> 850 Microservices
Cloud Provider API Outage
WAF Outage -> DisabledScalability Issues
Network is Unreliable
Autoscaling Keeps
Breaking
Large Customer OutageDelayed Features
DNS Resolution Errors
Expired Certificate
Regulatory Audit
Rolling Sev1 Outage on
Portal
Merger with
competitor
Misconfigured FW Rule Outage
Database Outage
Portal Retry Storm Outage
Orphaned Documentation
Corporate Reorg
Budget Freeze
Outsource overseas
development
Exposed Secrets on
GithuCode Freeze
b
Migration to New
CSP
Upgrade to Java
SE 12

21. Our systems become
more complex and
messy than we
remember them

22. So what does all
of this $&%*
have to do with
Security?

23. The
Normal
Condition
is to

24. We need failure
to Learn & Grow

25. How do we
typically discover
when our
security
measures fail?

26. Security incidents are not effective
measures of detection because at that
point it's already too late
26
Security Incidents

27. No System is inherently Secure
by Default, its Humans that
make them that way.

28. People
Operate
Differently
when they
expect things
to fail

31. @aaronrinehart @verica_io #chaosengineering
Chaos
Engineering

32. “Chaos Engineering is the discipline of
experimenting on a distributed system in
order to build confidence in the system’s
ability to withstand turbulent conditions”
Chaos Engineering

33. Chaos
Engineerin
gIs about establishing order
from Chaos

34. ● Define steady state
● Formulate hypothesis
● Outline methodology
● Identify blast radius
● Observability is key
● Readily abortable
Chaos Monkey
Story ● During Business Hours
● Born out of Netflix Cloud
Transformation
● Put well defined
problems in front of
engineers.
● Terminate VMs on
Random VPC Instances

35. Who is doing Chaos?

37. ExperimentationTesting vs.
Instrumenting Chaos

38. ● Formulate hypothesis
● Outline methodology
● Identify blast radius
● Observability is key
● Readily abortable
Chaos Pitfalls:Breaking things on Purpose
“I'm pretty sure I won’t have a
job very long if I break things
on purpose all day.”
-Casey Rosenthal
The purpose of Chaos Engineering
is NOT to “Break Things on
Purpose”.
If anything we are trying to “Fix
them on Purpose”!
Reference: Nora Jones 8 Traps of Chaos Engineering

39. SECURITY
CHAOS
ENGINEERING
39

40. “It worked in Star Wars but it
won’t work here”
Hope is Not
an Effective Strategy

41. “Understand your system and
where its security gaps are
before an adversary does”

42. WE OFTEN MISREMEMBER WHAT OUR
SYSTEMS REALLY ARE, AND AS A
RESULT THE OPPORTUNITY FOR
ACCIDENTS & MISTAKES INCREASES

43. Continuous
Security
Verification

44. Reduce Uncertainty by
Building Confidence
in how the system
actually functions

45. Use Cases

46. ● Incident Response
● Security Control Validation
● Security Observability
● Compliance Monitoring
Use Cases

47. “Response” is the problem
with incident response.
Incident
Response
47

48. We really don’t know very much
No matter how much we prepare...
Security Incidents are Subjective
Where? Why? Who?
What?How? 48

49. Post Mortem = Preparation
Flip the Model

50. 50
An Open Source Tool

51. • ChatOps Integration
• Configuration-as-Code
• Example Code & Open Framework
ChaoSlingr Product Features
• Serverless App in AWS
• 100% Native AWS
• Configurable Operational Mode &
Frequency
• Opt-In | Opt-Out Model

52. 52
An Example:
PortSlingr
Experiment
Hypothesis: A
Misconfigured or
Unauthorized Port
Change in AWS

53. Hypothesis: If someone accidentally or
maliciously introduced a misconfigured
port then we would immediately
detect, block, and alert on the event.
Alert
SOC?
Config
Mgmt?
Misconfigured
Port Injection
IR
Triage
Log
data?
Wait...
Firewall?

54. Alert
SOC?
Config
Mgmt?
Misconfigured
Port Injection
IR
Triage
Log
data?
Wait...
Firewall?
Test Opportunities

55. Stop looking for better
answers and start
asking better questions.
- John Allspaw

56. verica.io/sce-book
VERICA | CONTINUOUS VERIFICATION
Free copy to you
complements of Verica

57. verica.io/book
VERICA | CONTINUOUS VERIFICATION
Free complements of Verica
verica.io/sce-book