Security incident response is a reactive and chaotic exercise. What if it were possible to flip the scenario on its head? Security focused chaos engineering takes the approach of advancing the security incident response apparatus by reversing the postmortem and preparation phases. Contrary to Purple Team or Red Team game days, Security Chaos Engineering does not use threat actor tactics, techniques and procedures. It develops teams through unique configuration, cyber threat and user error scenarios that challenge responders to react to events outside their playbooks and comfort zones. Security Chaos Engineering allows incident response and product teams to derive new information about the state of security within their distributed systems that was previously unknown. Within this new paradigm of instrumentation where we proactively conduct “Pre-Incident” vs. “Post-Incident” reviews we are now able to more accurately measure how effective our security incident response teams, tools, skills, and procedures are during the manic of the Incident Response function. In this session Aaron Rinehart, the mind behind the first Open Source Security Chaos Engineering tool ChaoSlingr, will introduce how Security Chaos Engineering can be applied to create highly secure, performant, and resilient distributed systems.

1. @aaronrinehart @verica_io #chaosengineering

2. 2
About A.A.Ron
● CTO of Stealthy Startup
● Former Chief Security Architect
@UnitedHealth responsible for security
engineering strategy
● Led the DevOps and Open Source
Transformation at UnitedHealth Group
● Former (DOD, NASA, DHS, CollegeBoard )
● Frequent speaker and author on Chaos
Engineering & Security
● Pioneer behind Security Chaos Engineering
● Led ChaoSlingr team at UnitedHealth

3. CONFIDENTIAL

4. Security Precognition
@aaronrinehart @verica_io #chaosengineering

5. @aaronrinehart
“Resilience is the story of the
outage that never happened.”
- John Allspaw

6. 6
About A.A.Ron
● CTO of Stealthy Startup
● Former Chief Security Architect
@UnitedHealth responsible for security
engineering strategy
● Led the DevOps and Open Source
Transformation at UnitedHealth Group
● Former (DOD, NASA, DHS, CollegeBoard )
● Frequent speaker and author on Chaos
Engineering & Security
● Pioneer behind Security Chaos Engineering
● Led ChaoSlingr team at UnitedHealth

7. In this Session we will cover

8. Our systems have evolved
beyond human ability to
mentally model their behavior.
8

9. Our systems have evolved
beyond human ability to
mentally model their behavior.
9
everyone

11. Circuit Breaker Patterns
11
Continuous
Delivery
Distribute
d Systems
Blue/Green
Deployments
Cloud
Computing
Service Mesh
Containers
Immutable
InfrastructureInfracode
Continuous
Integration
Microservice
Architectures
API Auto Canaries
CI/CD
DevOps
Automation Pipelines
Complex?

12. 12
Mostly
Monolithic
Requires
Domain
Knowledge
Prevention
focused Poorly
Aligned
Defense
in Depth
Stateful in
nature
DevSecOps
not widely
adopted
Security?
Expert
Systems
Adversary
Focused

13. Simplify?

14. Software Only Increases in Complexity

15. Accidental
Complexity
Essential
Complexity
Software Complexity

16. Woods Theorem: “As the complexity of a system
increases, the accuracy of any single agent’s
own model of that system decreases”
- Dr. David Woods

17. How well do you really
understand how your system
works?

18. Difficult to Grok behavior

19. So what does all of this
have to do with Security?

20. Failure Happens.

21. Incidents & System
Outages are
Expensive

22. Security Incidents
are Subjective in
Nature

23. We really don't know
Where? Why? Who?
What?How?
very much

24. Lets face it, when outages happen…..

25. Teams spend too
much time
reacting to
outages instead
of building more
resilient systems.

26. “Response” is the
problem with Incident
Response

27. “Chaos Engineering is the discipline of
experimenting on a distributed system
in order to build confidence in the
system’s ability to withstand turbulent
conditions”

30. A PCh
AControl
AExperiment

31. Who is doing Chaos?

33. Security
Engineering

34. Security
Engineering

35. People Operate
Differently when they
expect things to fail

38. The Normal
Condition of
a Human &
Systems
they Build is
to
FAIL

39. We need failure
to Learn & Grow
39

40. Post Mortem = Preparation
Lets Flip the Model

41. Bring Order through Chaos

42. Use
Chaos Engineering
to initiate
Objective Feedback Loops
about Security
Effectiveness

43. Proactively Manage &
Measure Validate Runbooks
Measure Team Skills
Determine Control
Effectiveness
Learn new insights into
system behavior
Transfer knowledge
Build a learning culture

44. Testing vs. Experimentation

45. Security Crayon Differences
Noisy distributed system behavior
Not geared for Cascading Events
Point-in-time even if Automated
Performed by Security Teams with
Specialized skill sets

46. Security Chaos Differences
Distributed Systems Focus
Goal: Experimentation
Human Factors focused
Small Isolated Scope
Focus on Cascading Events
Performed by Mixed Engineering
Teams in Gameday
During business hours

47. 2018 Causes of Data Breaches

48. 2018 Causes of Data Breaches

49. 2018 Causes of Data Breaches

50. 2018 Causes of Data Breaches

51. ‘Human Error’, Root Cause, & Blame
Culture

52. Proactively
Manage & Measure

53. Continuous
SECURITY
Validation

54. Build Confidence
in
What
Actually Works

55. So how does it
work?

56. Stop looking for better answers and
start asking better questions.
- John Allspaw

57. What is the system actually doing?

58. What is the system actually doing?
Has it done this before?

59. What is the system actually doing?
Has it done this before?
Why is it behaving that way?

60. What is the system actually doing?
Has it done this before?
Why is it behaving that way?
What is it supposed to do next?

61. What is the system actually doing?
Has it done this before?
Why is it behaving that way?
What is it supposed to do next?
How did it get into this state?

62. How does My Security
Really Work?

63. What evidence do I
have to prove it?

64. 64
An Open Source
Tool

65. • ChatOps Integration
• Configuration-as-Code
• Example Code & Open Framework
ChaoSlingr Product Features
• Serverless App in AWS
• 100% Native AWS
• Configurable Operational Mode &
Frequency
• Opt-In | Opt-Out Model

66. Hypothesis: If someone accidentally or
maliciously introduced a misconfigured
port then we would immediately detect,
block, and alert on the event.
Alert
SOC?
Config
Mgmt?
Misconfigured
Port Injection
IR
Triage
Log
data?
Wait...
Firewall?

67. Result: Hypothesis disproved. Firewall did not detect
or block the change on all instances. Standard Port
AAA security policy out of sync on the Portal Team
instances. Port change did not trigger an alert and
log data indicated successful change audit.
However we unexpectedly learned the configuration
mgmt tool caught change and alerted the SoC.
Alert
SOC?
Config
Mgmt?
Misconfigured
Port Injection
IR
Triage
Log
data?
Wait...
Firewall?

68. More Experiment Examples
● Internet exposed
Kubernetes API
● Unauthorized Bad
Container Repo
● Unencrypted S3 Bucket
● Disable MFA
● Bad AWS Automated Block
Rule
● Software Secret Clear
Text Disclosure
● Permission collision in
Shared IAM Role Policy
● Disabled Service Event
Logging
● Introduce Latency on
Security Controls
● API Gateway Shutdown

69. Q&A
@aaronrinehart aaron@verica.io

70. Thank you!
@aaronrinehart aaron@verica.io

71. CONFIDENTIAL