Security today is customarily a reactive and chaotic exercise. In this session, we will introduce a new concept known as Security Chaos Engineering and how it can be applied to create highly secure, performant, and resilient distributed systems.

1. @aaronrinehart @verica_io #chaosengineering
Security Chaos
Engineering

2. In this Session we will cover

3. @aaronrinehart @verica_io #chaosengineering
● Combating Complexity in Software
● Chaos Engineering
● Resilience Engineering & Security
● Security Chaos Engineering
Areas Covered

4. 4
Aaron Rinehart, CTO, Founder
● Former Chief Security Architect
@UnitedHealth responsible for security
engineering strategy
● Led the DevOps and Open Source
Transformation at UnitedHealth Group
● Former (DOD, NASA, DHS, CollegeBoard )
● Frequent speaker and author on Chaos
Engineering & Security
● Pioneer behind Security Chaos Engineering
● Led ChaoSlingr team at UnitedHealth
@aaronrinehart @verica_io #chaosengineering
Verica

5. Incidents,Outages, &
Breaches are Costly

6. The Obvious Problem

7. Why do they
seem to be
happening more
often?

8. @aaronrinehart @verica_io #chaosengineering
Combating
Complexity in
Software

9. “The growth of complexity
in society has got ahead
of our understanding of
how complex systems
work and fail”
-Sydney Dekker

10. Our systems have evolved beyond human
ability to mentally model their behavior.
10

11. Our systems have evolved beyond human
ability to mentally model their behavior.
11
everyone
else

13. Circuit Breaker Patterns
Continuous
Delivery
Distributed
Systems
Blue/Green
Deployments
Cloud
Computing
Service Mesh
Containers
Immutable
Infrastructur
e
Infracod
e
Continuous
Integration
Microservice
Architectures
API Auto Canaries
CI/CD
DevOps
Automation Pipelines
Complex?

14. Mostly
Monolithic
Requires
Domain
Knowledge
Prevention
focused Poorly
Aligned
Defense
in Depth
Stateful in
nature
DevSecOps
not widely
adopted
Security?
Expert
Systems
Adversary
Focused

15. Simplify?

16. Software has
officially
taken over

17. Software Only Increases in Complexity

18. Accidental Essential
Software Complexity

19. “As the complexity of a system
increases, the accuracy of any single
agent’s own model of that system
decreases”
- Dr. David Woods
Woods Theorem:

20. What about my systems?

21. How well do you
really understand
how your system
works?

22. Systems
Engineering is
Messy
In Reality…….

23. In the
beginning...we
think it looks like

24. After a few
months….
Hard Coded Passwords
Identity Conflicts
Lead Software
Engineering finds a new
job at Google
New Security Tool
Refactor Pricing
300 Microservices Δ-> 850 Microservices
Cloud Provider API
Outage
WAF Outage -> DisabledScalability Issues
Network is Unreliable
Autoscaling Keeps
Breaking
Large Customer
Outage
Delayed Features
DNS Resolution
ErrorsExpired Certificate
Regulatory
Audit
Rolling Sev1
Outage on Portal
Code Freeze

25. Years?….
Hard Coded Passwords
Identity Conflicts
Lead Software Engineering
finds a new job at Google
New Security Tool
Refactor Pricing
300 Microservices Δ-> 4000 Microservices
Cloud Provider API Outage
Firewall Outage -> Disabled
Scalability Issues
Network is Unreliable
Autoscaling Keeps
Breaking
Large Customer
Outage
Delayed Features
DNS Resolution
Errors
Expired Certificate
Regulatory
Audit
Rolling Sev1 Outages on
Portal
Code Freeze
Hard Coded Passwords
Identity Conflicts
Lead Software Engineering
finds a new job at Google
New Security Tool
Refactor Pricing
300 Microservices Δ-> 850 Microservices
Cloud Provider API Outage
WAF Outage -> DisabledScalability Issues
Network is Unreliable
Autoscaling Keeps
Breaking
Large CustomerDelayed Features
DNS Resolution
ErrorsExpired Certificate
Regulatory
Audit
Rolling Sev1 Outage on
Portal
Merger with
competitor
Misconfigured FW Rule Outage
Database Outage
Portal Retry Storm
Outage
Orphaned Documentation
Corporate Reorg
Budget Freeze
Outsource overseas
development
Exposed Secrets on
GithuCode Freeze
b
Migration to New
CSP
Upgrade to Java
SE 12

26. Our systems become
more complex and
messy than we
remember them

27. Difficult to Mentally Model

28. Avoid Running in the Dark
@aaronrinehart @verica_io #chaosengineering

29. So what does all of
this $&%* have to
do with Security?

30. Failure Happens Alot

31. The
Normal
Condition
is to
FAIL

32. We need failure
to Learn & Grow
32

33. “things that have never
happened before happen all
the time”
–Scott Sagan “The Limits of Safety”

34. What happens when
our Security fails?

35. How do we typically
discover when our
security measures
fail?

36. Security
Incidents
Typically we dont find out our security is
failing until there is an security incident.

37. Vanishing
Traces
All we typically ever see is the
Footsteps in the Sand
-Allspaw
Logs, Stack Traces,
Alerts

38. Security incidents are
not effective measures of
detection
because at that point
it's already too late

39. What typically causes
our security to fail?

40. 2018 Causes of Data Breaches

41. 2018 Causes of Data Breaches

42. 2018 Causes of Data Breaches

43. 2018 Causes of Data Breaches

44. ‘Human-Error’, Root Cause, &
Blame Culture

45. No System is inherently Secure by
Default, its Humans that make them
that way.

46. People Operate Differently
when they expect things to
fail

49. @aaronrinehart @verica_io #chaosengineering
Chaos
Engineering

50. “Chaos Engineering is the discipline of
experimenting on a distributed system
in order to build confidence in the
system’s ability to withstand turbulent
conditions”
Chaos
Engineering

51. Who is doing Chaos?

54. “[Chaos Engineering is] empirical
rather than formal. We don’t use
models to understand what the
system should do. We run
experiments to learn what it does.”
- Michael T. Nygard

55. Use Chaos to Establish Order

56. Testing vs. Experimentation

57. ●
●
●
●
●
●
Properties of a
Chaos Experiment
Game Days allow you to perform
experiments with maximum visibility
and coverage from component
owners, support teams and product.
● Define steady state
● Formulate hypothesis
● Outline methodology
● Identify blast radius
● Observability is key
● Readily abortable

58. Developing a
Learning Culture
around Failure
● Safety as part of security
● Building safety margin
into systems
● Replace blame culture with
learning culture
● Telemetry, experimentation,
and instrumentation

59. ●
●
●
●
●
●
Chaos Engineering
Maturity
Despite what has been popularized on online
tech blogs you do not start off performing Chaos
Engineering on live production systems. There is
a maturity ramp to getting there.
● Validate Chaos Tools in
Lower Environment
● Develop Competency &
Confidence in Tooling
● Dry-run experiments
Warning: Still be careful in Non-Prod environments as you will be surprised what
hazards lie in Non-Prod. (Kafka Story)

60. ●
●
●
●
●
●
Chaos Monkey
Story
● During Business Hours
● Born out of Netflix Cloud
Transformation
● Put well defined problems
in front of engineers.
● Terminate VMs on
Random VPC Instances

61. ●
●
●
●
●
●
Chaos Engineering Pro-Tips
● Don’t perform an experiment
when you expect it to fail
● Auto Remediation of
Experiments will end in a
fiery Hell!
● Transparency is a Must
● Webcast & Record
GameDays
● The process of creating the
experiment and sharing the
learnings is the
highest-value of Chaos
Engineering
● Chaos Engineering Goal:
Share Team Mental Models
is of High Importance

62. ●
●
●
●
●
●
Chaos Pitfalls: Auto-Remediation
“…an operator will only be able to generate successful new
strategies for unusual situations if he has an adequate
knowledge of the process.”
“ Long term knowledge develops only through use and
feedback about its effectiveness.”
— Lisanne Bainbridge, The Ironies of Automation (1983)
Bring context or chase down
vulnerabilities for the service
owner instead of automating
fixes as this leads to a Fiery
Hell!
Reference: Nora Jones 8 Traps of Chaos Engineering

63. ●
●
●
●
●
●
Chaos Pitfalls:Breaking things on Purpose
“I'm pretty sure
I won’t have a job
very long if I
break things on
purpose all day.”
-Casey Rosenthal
The purpose of Chaos Engineering is NOT
to “Break Things on Purpose”.
If anything we are trying to “Fix them on
Purpose”!
Reference: Nora Jones 8 Traps of Chaos Engineering

64. ●
●
●
●
●
●
Chaos Engineering
Operational Models
● Organization-Wide Chaos Engineering
Team
● Provide a Chaos Engineering Solution for
Teams to Consume
● CentralTeam runs periodic Chaos
Experiments as a Service
● Provide SREs with Chaos Toolsets
“At Netflix Chaos Engineering
was always meant to be a
tools practice for SREs”
- Casey Rosenthal

65. ●
●
●
●
●
●
GameDay Exercises
● 2-4 hrs in Length
● Diverse Cross Functional Group of
Engineers
● Focused on Increasing Resilience
● Used for Manual Chaos
Engineering
● Great Introduction to Chaos
Engineering
Recommendations
● Use GameDays for New Chaos
Experiments
● Use GameDays for Initial
Experiment Deployment on New
Targets
● Use GameDays for Proving New
Chaos Engineering Tools
● Get Everyone in the Same Location

66. ● Define steady state
● Formulate hypothesis
● Outline methodology
● Identify blast radius
● Observability is key
● Readily abortable
Experiment Lifecycle
1
Perform a GameDay
Exercise
Plan, Schedule, and Run a
GameDay Exercise for
New Experiments
Validate Experiment
Hypothesis
Goal: Validate
experiment ran
successfully and that
the results are credible.
2
Remediate Findings &
Repeat Experiment
If hypothesis failed for
the experiment. Develop
and remediate list of
findings. Once
remediated, repeat
experiment
3
Once Successful:
Automate Experiment
Once the experiment has
been proved to run
successfully validating
your hypothesis you can
now automate the
experiment runs
periodically..
4

67. GameDays: The Basics
Plan &
Organize
GameDay
Exercise
Execute
Live
GameDay
Operations
Automate &
Evangelize
Results & Take
Action
Chaos
Experiment
Develop &
Evaluate
Conduct
Pre-Incident
Review

68. @aaronrinehart @verica_io #chaosengineering
Security
Chaos
Engineering

69. “The discipline of instrumentation, identification,
and remediation of failure within security controls
through proactive experimentation to build
confidence in the system's ability to defend
against malicious conditions in production.”
Security Chaos Engineering is...

70. Continuous
Security
Verification

71. Proactively
Manage & Measure

72. Reduce Uncertainty by
Building Confidence

73. Build Confidence
in
What Actually Works

74. @aaronrinehart @verica_io #chaosengineering
Security Chaos
Engineering
Use Cases

75. Security Incidents
are Subjective in
Nature

76. We really don't know
Where? Why? Who?
What?How?
very much

77. “Response” is the
problem with Incident
Response

78. Lets face it, when outages
happen…..
Teams spend too much time
reacting to outages instead
of building more resilient
systems.

79. Post Mortem = Preparation
Lets Flip the Model

81. Solution
Architecture
“More men(people) die from
their remedies not their
illnesses”
- Jean-Baptiste Poquelin

82. 87
Solutions Architecture
needs reinvention
Patterns never worked
Ivory Tower Architecture

83. Security
Control
Validation

84. 90
An Open Source
Tool

85. • ChatOps Integration
• Configuration-as-Code
• Example Code & Open Framework
ChaoSlingr Product Features
• Serverless App in AWS
• 100% Native AWS
• Configurable Operational Mode &
Frequency
• Opt-In | Opt-Out Model

86. Hypothesis: If someone accidentally or
maliciously introduced a misconfigured
port then we would immediately detect,
block, and alert on the event.
Alert
SOC?
Config
Mgmt?
Misconfigured
Port Injection
IR
Triage
Log
data?
Wait...
Firewall?

87. Result: Hypothesis disproved. Firewall did not detect
or block the change on all instances. Standard Port
AAA security policy out of sync on the Portal Team
instances. Port change did not trigger an alert and
log data indicated successful change audit.
However we unexpectedly learned the configuration
mgmt tool caught change and alerted the SoC.
Alert
SOC?
Config
Mgmt?
Misconfigured
Port Injection
IR
Triage
Log
data?
Wait...
Firewall?

88. Stop looking for better
answers and start asking
better questions.
- John Allspaw

89. What is the system actually doing?
Has it done this before?
Why is it behaving that way?
What is it supposed to do next?
How did it get into this state?

90. How does My Security
Really Work?

91. What evidence do I
have to prove it?

92. Cloud Security
Readiness
● Verify Saas Security
Controls
● Verify Cloud Native
Controls
● Verify Security
Configuration

93. Security
Observability
Monitoring Logging
Tracing Visualization

94. Security Log
Pipelines Monitoring
Logging
Tracing
Visualization

95. Improve Value of
Security Log Data
● How valuable is your log
data?
● When do we ever assess
this?
● We dont know our logs
are shit until we
absolutely need them
● Proactively determine
quality of log data
around experiments

96. Create Objective Feedback
Loops about Security
Effectiveness

97. How does Security Chaos Engineering
differ from Red Teaming, Purple
Teaming or Pen Testing?
Security
Crayons

98. ● Distributed Systems Focus
● Goal: Experimentation
● Human Factors focused
● Small Isolated Scope
● Focus on Cascading Events
● Performed by Mixed Engineering Teams
in Gameday
● During business hours
Differences in Scope, Focus, and Method

99. Q&A
@aaronrinehart aaron@verica.io