This document discusses using chaos engineering and security chaos engineering techniques to proactively test systems and security controls. It notes that as software systems increase in complexity, it becomes difficult for teams to mentally model how their systems work. Rather than waiting for security incidents to discover failures, chaos engineering experiments intentionally induce failures to build confidence in a system's resilience. The document provides examples of security chaos engineering use cases like validating security controls and improving incident response processes. It argues this approach reduces uncertainty and helps teams continuously harden their systems and security.

1. NOVEMBER 6, 2019

3. NOVEMBER 6, 2019
Security Pre-Cognition
DevSecOps &
Chaos Engineering

4. @aaronrinehart @verica_io #chaosengineering
Security Precognition
DevSecOps & Chaos Engineering

5. In this Session we will cover

6. @aaronrinehart @verica_io #chaosengineering
● Combating Complexity in Software
● Chaos Engineering
● Resilience Engineering & Security
● Security Chaos Engineering
Areas Covered

7. 7
Aaron Rinehart, CTO, Founder
● Former Chief Security Architect
@UnitedHealth responsible for security
engineering strategy
● DevOps and Open Source Transformation
at UnitedHealth Group
● Former (DOD, NASA, DHS, CollegeBoard )
● Frequent speaker and author on Chaos
Engineering & Security
● Pioneer behind Security Chaos Engineering
● Led ChaoSlingr team at UnitedHealth
@aaronrinehart @verica_io #chaosengineering

8. Incidents,Outages, &
Breaches are Costly

9. An Obvious
Problem

10. Why do they
seem to be
happening more
often?

11. @aaronrinehart @verica_io #chaosengineering
Combating
Complexity in
Software

12. “The growth of complexity
in society has got ahead
of our understanding of
how complex systems work
and fail”
-Sydney Dekker

13. Our systems have evolved beyond human
ability to mentally model their behavior.

14. everyone else
Our systems have evolved beyond human
ability to mentally model their behavior.

16. Circuit Breaker Patterns
Continuous Delivery
Distributed
Systems
Blue/Green
Deployments
Cloud
ComputingService Mesh
Containers
Immutable
Infrastructure
Infracode
Continuous Integration
Microservice
Architectures
API Auto Canaries
CI/CDDevOps
Automation Pipelines
Complex?

17. Mostly
Monolithic
Requires Domain
Knowledge
Prevention
focused Poorly Aligned
Defense in
Depth
Stateful in
nature
DevSecOps not
widely adopted
Security?
Expert
Systems
Adversary Focused

18. Simplify?

19. Software has
officially
taken over

20. Software Only Increases in Complexity

21. Accidental Essential
Software Complexity

22. “As the complexity of a system
increases, the accuracy of any single
agent’s own model of that system
decreases”
- Dr. David Woods
Woods Theorem:

23. What does this have to do
with my systems?

24. Question - How well
do you really
understand how
your system works?

25. Systems
Engineering is
Messy
In Reality…….

26. In the
beginning...we
think it looks like

27. After a few
months….
Hard Coded Passwords
Identity Conflicts
Lead Software
Engineering finds a new
job at Google
New Security Tool
Refactor Pricing
300 Microservices Δ-> 850 Microservices
Cloud Provider API
Outage
WAF Outage -> DisabledScalability Issues
Network is Unreliable
Autoscaling Keeps
Breaking
Large CustomerDelayed Features
DNS Resolution
ErrorsExpired Certificate
Regulatory
Audit
Rolling Sev1
Outage on Portal
Code Freeze

28. Years?….
Hard Coded Passwords
Identity Conflicts
Lead Software Engineering
finds a new job at Google
New Security Tool
Refactor Pricing
300 Microservices Δ-> 4000 Microservices
Cloud Provider API Outage
Firewall Outage -> Disabled
Scalability Issues
Network is Unreliable
Autoscaling Keeps
Breaking
Large Customer
Outage
Delayed Features
DNS Resolution
Errors
Expired Certificate
Regulatory
Audit
Rolling Sev1 Outages on
Portal
Code Freeze
Hard Coded Passwords
Identity Conflicts
Lead Software Engineering
finds a new job at Google
New Security Tool
Refactor Pricing
300 Microservices Δ-> 850 Microservices
Cloud Provider API Outage
WAF Outage -> DisabledScalability Issues
Network is Unreliable
Autoscaling Keeps
Breaking
Large CustomerDelayed Features
DNS Resolution
ErrorsExpired Certificate
Regulatory
Audit
Rolling Sev1 Outage on
Portal
Merger with
competitor
Misconfigured FW Rule Outage
Database Outage
Portal Retry Storm
Outage
Orphaned Documentation
Corporate Reorg
Budget Freeze
Outsource overseas
development
Exposed Secrets on
GithuCode Freeze
b
Migration to New
CSP
Upgrade to Java
SE 12

29. Our systems become
more complex and
messy than we
remember them

30. Difficult to Mentally Model

31. Avoid Running in the Dark
@aaronrinehart @verica_io #chaosengineering

32. So what does all of
this $&%* have to
do with Security?

33. Failure Happens Alot

34. The
Normal
Condition
is to

35. We need failure
to Learn & Grow

36. “things that have never
happened before happen all the
time”
–Scott Sagan “The Limits of Safety”

37. How do we typically
discover when our
security measures
fail?

38. Security
Incidents
Typically we dont find out our security is
failing until there is an security incident.

39. Vanishing
Traces
All we typically ever see is the
Footsteps in the Sand
-Allspaw
Logs, Stack Traces,
Alerts

40. Security incidents are
not effective measures of
detection
because at that point
it's already too late

41. What typically causes
our security to fail?

42. 2018 Causes of Data Breaches

43. 2018 Causes of Data Breaches

44. 2018 Causes of Data Breaches

45. 2018 Causes of Data Breaches

46. ‘Human-Error’, Root Cause, &
Blame Culture

47. No System is inherently Secure by
Default, its Humans that make them
that way.

48. Considerations in
building a Learning Culture
Cognitive Load, Bias, and Blame

49. People Operate Differently
when they expect things to
fail

52. @aaronrinehart @verica_io #chaosengineering
Chaos
Engineering

53. “Chaos Engineering is the discipline of
experimenting on a distributed system
in order to build confidence in the
system’s ability to withstand turbulent
conditions”
Chaos Engineering

54. Who is doing Chaos?

57. Use Chaos to Establish Order

58. Testing vs. Experimentation

59. ● Define steady state
● Formulate hypothesis
● Outline methodology
● Identify blast radius
● Observability is key
● Readily abortable
Chaos Monkey
Story
● During Business Hours
● Born out of Netflix Cloud
Transformation
● Put well defined problems
in front of engineers.
● Terminate VMs on
Random VPC Instances

60. ● Define steady state
● Formulate hypothesis
● Outline methodology
● Identify blast radius
● Observability is key
● Readily abortable
Chaos Pitfalls:Breaking things on Purpose
“I'm pretty sure
I won’t have a job
very long if I
break things on
purpose all day.”
-Casey Rosenthal
The purpose of Chaos Engineering is NOT
to “Break Things on Purpose”.
If anything we are trying to “Fix them on
Purpose”!
Reference: Nora Jones 8 Traps of Chaos Engineering

61. @aaronrinehart @verica_io #chaosengineering
Security
Chaos
Engineering

62. Continuous
Security
Verification

63. Proactively
Manage & Measure

64. Reduce Uncertainty by
Building Confidence in
how the system
actually functions

65. @aaronrinehart @verica_io #chaosengineering
Security Chaos
Engineering
Use Cases

66. @aaronrinehart @verica_io #chaosengineering
● Incident Response
● Solutions Architecture
● Security Control Validation
● Security Observability
● Continuous Verification
● Compliance Monitoring
Use Cases

67. Incident
Response

68. Security Incidents
are Subjective in
Nature

69. We really don't know
Where? Why? Who?
What?How?
very much

70. “Response” is the problem
with Incident Response

71. Lets face it, when outages
happen…..
Teams spend too much time
reacting to outages instead
of building more resilient
systems.

74. Solution
Architecture
“More men(people) die from
their remedies not their
illnesses”
- Jean-Baptiste Poquelin

75. Solutions Architecture
needs reinvention
Patterns never worked
Ivory Tower Architecture

76. Security
Control
Validation

77. Create Objective Feedback
Loops about Security
Effectiveness

78. An Open Source
Tool

80. Hypothesis: If someone accidentally or maliciously introduced a
misconfigured port then we would immediately detect, block, and
alert on the event.
Alert
SOC?
Config
Mgmt?
Misconfigured
Port Injection
IR
Triage
Log
data?
Wait...
Firewall?

81. Result: Hypothesis disproved. Firewall did not detect or block the change on all instances.
Standard Port AAA security policy out of sync on the Portal Team instances. Port change
did not trigger an alert and log data indicated successful change audit.
However we unexpectedly learned the configuration mgmt tool caught change and alerted
the SoC.
Alert
SOC?
Config
Mgmt?
Misconfigured
Port Injection
IR
Triage
Log
data?
Wait...
Firewall?

82. Stop looking for better
answers and start asking
better questions.
- John Allspaw

85. NOVEMBER 6, 2019
Security Pre-Cognition
DevSecOps &
Chaos Engineering