The speed and scale of complex system operations within cloud-driven architectures make them extremely difficult for humans to mentally model their behavior. This often results in unpredictable and catastrophic outcomes that become costly when unexpected security incidents occur. There is a need to realign the actual state of operational security measures in order to maintain an acceptable level of confidence that our security actually works when we need it to. As an alternative to simply reacting to failures, the security industry has been overlooking valuable chances to further understand and nurture ‘accidents’ or ‘mistakes’ as opportunities to proactively strengthen system resilience. Chaos Engineering allows us to proactively expose the failures, build resilient systems, and develop an "Applied Security" model to minimize the impact of failures. Chaos Engineering allows for security teams to proactively experiment and derive new information about underlying factors that were previously unknown. This is done by developing live fire exercises that can be measured, managed, and automated. Contrary to Red/Purple Team exercises, chaos engineering does not use threat actor or adversarial tactics, techniques and procedures. As far as we know it Chaos Engineering is the only proactive mechanism for detecting availability and security incidents before they happen. We proactively introduce turbulent conditions, faults, and failures into our systems to determine the conditions by which our security will fail before it actually does. In this session we will introduce a new concept known as Security Chaos Engineering and how it can be applied to create highly secure, performant, and resilient distributed systems.

1. @aaronrinehart @verica_io #chaosengineering
AllDayDevOps
2020 Spring Edition
Navigating the DevSecOps
App-ocolypse
w/ Security Chaos
Engineering

2. In this Session we will cover

3. @aaronrinehart @verica_io #chaosengineering
● Combating Complexity in Software
● Chaos Engineering
● Resilience Engineering & Security
● Security Chaos Engineering
Areas Covered

4. 4
Aaron Rinehart, CTO, Founder
● Former Chief Security Architect
@UnitedHealth
● Former DoD, NASA Safety & Reliability
Engineering
● Frequent speaker and author on Chaos
Engineering & Security
● O’Reilly Author: Chaos Engineering, Security
Chaos Engineering Books
● Pioneer behind Security Chaos Engineering
● Led ChaoSlingr team at UnitedHealth
@aaronrinehart @verica_io #chaosengineering

5. Incidents,Outages, &
Breaches are Costly

6. An Obvious
Problem

7. Why do they
seem to be
happening more
often?

8. @aaronrinehart @verica_io #chaosengineering
Combating
Complexity in
Software

9. “The growth of complexity
in society has got ahead
of our understanding of
how complex systems
work and fail”
-Sydney Dekker

10. Our systems have evolved beyond human
ability to mentally model their behavior.
10

11. 11
everyone else
Our systems have evolved beyond human
ability to mentally model their behavior.

13. Circuit Breaker Patterns
Continuous Delivery
Distributed
Systems
Blue/Green
Deployments
Cloud
ComputingService Mesh
Containers
Immutable
Infrastructure
Infracode
Continuous
Integration
Microservice
Architectures
API Auto Canaries
CI/CDDevOps
Automation Pipelines
Complex?

14. Mostly
Monolithic
Requires
Domain
Knowledge
Prevention
focused Poorly Aligned
Defense in
Depth
Stateful in
nature
DevSecOps not
widely adopted
Security?
Expert
Systems
Adversary Focused

15. Simplify?

16. Software has
officially
taken over

17. Software Only Increases in Complexity

18. Accidental Essential
Software Complexity

19. “As the complexity of a system
increases, the accuracy of any single
agent’s own model of that system
decreases”
- Dr. David Woods
Woods Theorem:

20. What does this have to do
with my systems?

21. Question - How well
do you really
understand how
your system works?

22. Systems
Engineering is
Messy
In Reality…….

23. In the
beginning...we
think it looks like

24. After a few
months….
Hard Coded Passwords
Identity Conflicts
Lead Software
Engineering finds a new
job at Google
New Security Tool
Refactor Pricing
300 Microservices Δ-> 850 Microservices
Cloud Provider API
Outage
WAF Outage -> DisabledScalability Issues
Network is Unreliable
Autoscaling Keeps
Breaking
Large CustomerDelayed Features
DNS Resolution
ErrorsExpired Certificate
Regulatory
Audit
Rolling Sev1
Outage on Portal
Code Freeze

25. Years?….
Hard Coded Passwords
Identity Conflicts
Lead Software Engineering
finds a new job at Google
New Security Tool
Refactor Pricing
300 Microservices Δ-> 4000 Microservices
Cloud Provider API Outage
Firewall Outage -> Disabled
Scalability Issues
Network is Unreliable
Autoscaling Keeps
Breaking
Large Customer
Outage
Delayed Features
DNS Resolution
Errors
Expired Certificate
Regulatory
Audit
Rolling Sev1 Outages on
Portal
Code Freeze
Hard Coded Passwords
Identity Conflicts
Lead Software Engineering
finds a new job at Google
New Security Tool
Refactor Pricing
300 Microservices Δ-> 850 Microservices
Cloud Provider API Outage
WAF Outage -> DisabledScalability Issues
Network is Unreliable
Autoscaling Keeps
Breaking
Large CustomerDelayed Features
DNS Resolution
ErrorsExpired Certificate
Regulatory
Audit
Rolling Sev1 Outage on
Portal
Merger with
competitor
Misconfigured FW Rule Outage
Database Outage
Portal Retry Storm
Outage
Orphaned Documentation
Corporate Reorg
Budget Freeze
Outsource overseas
development
Exposed Secrets on
GithuCode Freeze
b
Migration to New
CSP
Upgrade to Java
SE 12

26. Our systems become
more complex and
messy than we
remember them

27. Difficult to Mentally Model

28. So what does all of
this $&%* have to
do with Security?

29. Failure Happens Alot

30. The
Normal
Condition
is to
FAIL

31. We need failure
to Learn & Grow
31

32. “things that have never
happened before happen all the
time”
–Scott Sagan “The Limits of Safety”

33. How do we typically
discover when our
security measures
fail?

34. Security
Incidents
Typically we dont find out our security is
failing until there is an security incident.

35. Vanishing
Traces
All we typically ever see is the
Footsteps in the Sand
-Allspaw
Logs, Stack Traces,
Alerts

36. Security incidents are
not effective measures of
detection
because at that point
it's already too late

37. No System is inherently Secure by
Default, its Humans that make them
that way.

38. People Operate Differently
when they expect things to
fail

41. @aaronrinehart @verica_io #chaosengineering
Chaos
Engineering

42. “Chaos Engineering is the discipline of
experimenting on a distributed system
in order to build confidence in the
system’s ability to withstand turbulent
conditions”
Chaos Engineering

43. Who is doing Chaos?

46. Use Chaos to Establish Order

47. Testing vs. Experimentation

48. ● Define steady state
● Formulate hypothesis
● Outline methodology
● Identify blast radius
● Observability is key
● Readily abortable
Chaos Monkey
Story
● During Business Hours
● Born out of Netflix Cloud
Transformation
● Put well defined problems
in front of engineers.
● Terminate VMs on
Random VPC Instances

49. ● Define steady state
● Formulate hypothesis
● Outline methodology
● Identify blast radius
● Observability is key
● Readily abortable
Chaos Pitfalls:Breaking things on Purpose
“I'm pretty sure
I won’t have a job
very long if I
break things on
purpose all day.”
-Casey Rosenthal
The purpose of Chaos Engineering is NOT
to “Break Things on Purpose”.
If anything we are trying to “Fix them on
Purpose”!
Reference: Nora Jones 8 Traps of Chaos Engineering

50. @aaronrinehart @verica_io #chaosengineering
Security
Chaos
Engineering

51. Continuous
Security
Verification

52. Proactively
Manage & Measure

53. Reduce Uncertainty by
Building Confidence in
how the system
actually functions

54. @aaronrinehart @verica_io #chaosengineering
Security Chaos
Engineering
Use Cases

55. @aaronrinehart @verica_io #chaosengineering
● Incident Response
● Solutions Architecture
● Security Control Validation
● Security Observability
● Continuous Verification
● Compliance Monitoring
Use Cases

56. Incident Response

57. Security Incidents
are Subjective in
Nature

58. We really don't know
Where? Why? Who?
What?How?
very much

59. Lets face it, when outages
happen…..
Teams spend too much time
reacting to outages instead
of building more resilient
systems.

60. Post Mortem = Preparation
Lets Flip the Model

62. 62
An Open Source
Tool

63. • ChatOps Integration
• Configuration-as-Code
• Example Code & Open Framework
ChaoSlingr Product Features
• Serverless App in AWS
• 100% Native AWS
• Configurable Operational Mode &
Frequency
• Opt-In | Opt-Out Model

64. Hypothesis: If someone accidentally or
maliciously introduced a misconfigured
port then we would immediately detect,
block, and alert on the event.
Alert
SOC?
Config
Mgmt?
Misconfigured
Port Injection
IR
Triage
Log
data?
Wait...
Firewall?

65. Result: Hypothesis disproved. Firewall did not detect
or block the change on all instances. Standard Port
AAA security policy out of sync on the Portal Team
instances. Port change did not trigger an alert and
log data indicated successful change audit.
However we unexpectedly learned the configuration
mgmt tool caught change and alerted the SoC.
Alert
SOC?
Config
Mgmt?
Misconfigured
Port Injection
IR
Triage
Log
data?
Wait...
Firewall?

66. Stop looking for better
answers and start asking
better questions.
- John Allspaw

67. cutt.ly/verica-book
VERICA | CONTINUOUS VERIFICATION
Free copy mailed to you
complements of Verica