This document discusses security chaos engineering and how it can be used to proactively test systems and security controls. It notes that as systems increase in complexity, it becomes harder to fully understand their behavior. Security chaos engineering involves experimenting on systems to build confidence in their ability to withstand failures. Some key points made include: - Conducting controlled experiments simulating security incidents to validate response plans and identify weaknesses - Taking a proactive approach focused on learning rather than blame to improve security over time - Using an open source tool called ChaoSlingr to automate security chaos experiments across distributed systems

1. @aaronrinehart @verica_io #chaosengineering

2. @aaronrinehart @verica_io #chaosengineering

3. CONFIDENTIAL

4. Security Precognition
@aaronrinehart @verica_io #chaosengineering

5. @aaronrinehart
“Resilience is the story of the
outage that never happened.”
- John Allspaw

6. 6
About A.A.Ron
● CTO of Stealthy Startup
● Former Chief Security Architect
@UnitedHealth responsible for security
engineering strategy
● Led the DevOps and Open Source
Transformation at UnitedHealth Group
● Former (DOD, NASA, DHS, CollegeBoard )
● Frequent speaker and author on Chaos
Engineering & Security
● Pioneer behind Security Chaos Engineering
● Led ChaoSlingr team at UnitedHealth

7. In this Session we will cover

8. Our systems have evolved
beyond human ability to
mentally model their behavior.
8

9. Our systems have evolved
beyond human ability to
mentally model their behavior.
9
everyone

11. Circuit Breaker Patterns
11
Continuous
Delivery
Distribute
d Systems
Blue/Green
Deployments
Cloud
Computing
Service Mesh
Containers
Immutable
InfrastructureInfracode
Continuous
Integration
Microservice
Architectures
API Auto Canaries
CI/CD
DevOps
Automation Pipelines
Complex?

12. 12
Mostly
Monolithic
Requires
Domain
Knowledge
Prevention
focused Poorly
Aligned
Defense
in Depth
Stateful in
nature
DevSecOps
not widely
adopted
Security?
Expert
Systems
Adversary
Focused

13. Simplify?

14. Software Only Increases in Complexity

15. Accidental
Complexity
Essential
Complexity
Software Complexity

16. Woods Theorem: “As the complexity of a system
increases, the accuracy of any single agent’s
own model of that system decreases”
- Dr. David Woods

17. How well do you really
understand how your system
works?

18. Difficult to Grok behavior

19. So what does all of this
have to do with Security?

20. Failure Happens.

21. Incidents & System
Outages are
Expensive

22. Security Incidents
are Subjective in
Nature

23. We really don't know
Where? Why? Who?
What?How?
very much

24. Lets face it, when outages happen…..

25. Teams spend too
much time
reacting to
outages instead
of building more
resilient systems.

26. “Response” is the
problem with Incident
Response

27. “Chaos Engineering is the discipline of
experimenting on a distributed system
in order to build confidence in the
system’s ability to withstand turbulent
conditions”

30. A PCh
AControl
AExperiment

31. Who is doing Chaos?

33. Security
Engineering

34. Security
Engineering

35. People Operate
Differently when they
expect things to fail

38. The Normal
Condition of
a Human &
Systems
they Build is
to
FAIL

39. We need failure
to Learn & Grow
39

40. Post Mortem = Preparation
Lets Flip the Model

41. Bring Order through Chaos

42. Use
Chaos Engineering
to initiate
Objective Feedback Loops
about Security
Effectiveness

43. Proactively Manage &
Measure Validate Runbooks
Measure Team Skills
Determine Control
Effectiveness
Learn new insights into
system behavior
Transfer knowledge
Build a learning culture

44. Testing vs. Experimentation

45. Security Crayon Differences
Noisy distributed system behavior
Not geared for Cascading Events
Point-in-time even if Automated
Performed by Security Teams with
Specialized skill sets

46. Security Chaos Differences
Distributed Systems Focus
Goal: Experimentation
Human Factors focused
Small Isolated Scope
Focus on Cascading Events
Performed by Mixed Engineering
Teams in Gameday
During business hours

47. 2018 Causes of Data Breaches

48. 2018 Causes of Data Breaches

49. 2018 Causes of Data Breaches

50. 2018 Causes of Data Breaches

51. ‘Human Error’, Root Cause, & Blame
Culture

52. Proactively
Manage & Measure

53. Continuous
SECURITY
Validation

54. Build Confidence
in
What
Actually Works

55. So how does it
work?

56. Stop looking for better answers and
start asking better questions.
- John Allspaw

57. What is the system actually doing?

58. What is the system actually doing?
Has it done this before?

59. What is the system actually doing?
Has it done this before?
Why is it behaving that way?

60. What is the system actually doing?
Has it done this before?
Why is it behaving that way?
What is it supposed to do next?

61. What is the system actually doing?
Has it done this before?
Why is it behaving that way?
What is it supposed to do next?
How did it get into this state?

62. How does My Security
Really Work?

63. What evidence do I
have to prove it?

64. 64
An Open Source
Tool

65. • ChatOps Integration
• Configuration-as-Code
• Example Code & Open Framework
ChaoSlingr Product Features
• Serverless App in AWS
• 100% Native AWS
• Configurable Operational Mode &
Frequency
• Opt-In | Opt-Out Model

66. Hypothesis: If someone accidentally or
maliciously introduced a misconfigured
port then we would immediately detect,
block, and alert on the event.
Alert
SOC?
Config
Mgmt?
Misconfigured
Port Injection
IR
Triage
Log
data?
Wait...
Firewall?

67. Result: Hypothesis disproved. Firewall did not detect
or block the change on all instances. Standard Port
AAA security policy out of sync on the Portal Team
instances. Port change did not trigger an alert and
log data indicated successful change audit.
However we unexpectedly learned the configuration
mgmt tool caught change and alerted the SoC.
Alert
SOC?
Config
Mgmt?
Misconfigured
Port Injection
IR
Triage
Log
data?
Wait...
Firewall?

68. More Experiment Examples
● Internet exposed
Kubernetes API
● Unauthorized Bad
Container Repo
● Unencrypted S3 Bucket
● Disable MFA
● Bad AWS Automated Block
Rule
● Software Secret Clear
Text Disclosure
● Permission collision in
Shared IAM Role Policy
● Disabled Service Event
Logging
● Introduce Latency on
Security Controls
● API Gateway Shutdown

69. Q&A
@aaronrinehart aaron@verica.io

70. Thank you!
@aaronrinehart aaron@verica.io

71. CONFIDENTIAL