The document outlines a comprehensive guide on conducting effective security operations center (SOC) response simulations, detailing the rationale, organization, and execution processes involved. It emphasizes the importance of validating tools and procedures through controlled attacks, establishing clear roles within the teams, and detailing the critical concepts and major steps necessary for effective exercises. Additionally, it highlights the need for post-exercise reviews to document findings and improve future security operations.

1. Exercise Your SOC
How to run an effective SOC response simulation
04/29/2018
Brian Andrzejewski, Senior Cyber Security Engineer
Cyber Defense Branch, Information Security Division

2. Disclaimers
• My personal views and opinions may not
represent the position(s) of my employers
(USCIS, DHS, or Federal Government)
• Mention of any product names in this talk are not
a government endorsement.
• Questions? Raise hand to ask mid-stream!
UNCLASSIFED 2

3. About Organization
• USCIS: World’s largest immigration agency
– Annually:
• 8.5M applications received
• 1M Permanent Residents
• 730,000 new Citizens
• 7,200 military personnel
– Fee-funded
• Over $4B in annual revenue
• 17,000+ employees in over 200+ locations
– We are a leading Federal agency in the cloud!
UNCLASSIFIED 3

4. About Me
• Cyber Defense - Lead Federal security engineer
– Specialized in AppSec, DevSec, & CloudSec
• Assess systems & designs for weaknesses & vulns
• Build risk mitigations and “secure by default” profiles
• Embrace “infrastructure-as-code” & rugged DevOps
– IPT component rep for DHS Cloud Migration,
DevSecOps, & Cyber Threat Intel
– Advise Fed agencies on 2 yrs of InfoSec lessons
learned in using Cloud, Agile, & DevOps methods
• Prior life:
– US Air Force / DoD Cyber Crime Center (DC3)
• CNCI-5 / ESSA Program: Connect Fed Cyber Centers w/ STIX & TAXII
• Workforce Dev: DC3 Digital Forensics Challenge / U.S. Cyber Challenge
• Ops: DIBNET, Tool Assessment, Program Manager, Security Assessment
– IT Dev & Ops: Commercial Healthcare, Consulting, & Academia
UNCLASSIFIED 4

5. Personal Philosophies
“Doveryai, no proveryai!”
(trust, but verify)
- Suzanne Massie
“Attack is the secret of defense;
defense is the planning of an attack.”
– Sun Tzu
Burn down your technical debt
before it burns you.
UNCLASSIFIED 5

6. Our Security Challenges
UNCLASSIFIED 6
Security: we are [usually] the last to know… and first to respond.

7. Acronym Soup
• APT: Advanced Persistent Threat
• IR: Incident Response (typically SOC)
• TTPs: Tools, Techniques, & Procedures
• SOC: Security Ops Center Response team
UNCLASSIFIED 7

8. Why Do a SOC Exercise?
Validate your implemented TTPs for
SOC operate as expected
Test effectiveness of an attack
[in a controlled execution – on prod]
Finding your actual defense thresholds
UNCLASSIFIED 8

9. SOC Exercise Endgame
• Proactive training for attack patterns against your
organization’s TTPs [before advisory does]
• Learning together (red + blue) where your defenses &
processes are good, bad, and ugly and craft events together
• Determining where to focus [highly limited] resources post-
exercise to reduce risks of future compromise
UNCLASSIFIED 9
Avoid becoming a
[insert major news here]
article.

10. SOC Exercise: Team Roles
• Red Leader
– Has official comms to CISO on exercise execution
– Leads team through exercise build, run, & closeout
• Gold Leader
– Monitors “chatter” for exercise IR detection response
– Records outcomes to events executed
• Execution Team
– 1-2 Red Teamers specializing in the attack
– 1-2 Blue Teamers specializing in defense config and
its possible unexpected behaviors
UNCLASSIFIED 10

11. SOC Exercise: Critical Concepts
• Test hypotheses, not
destroy targets
– Objective testing of TTPs
– Assume professionalism for
targets involved
– Use dummy data &
modified, benign attacks
– Let results show actual
security posture
UNCLASSIFIED 11
Question
Recon
Test Hypothesis
Conclusion

12. SOC Exercise: Critical Concepts
• Purpose is to be detected
–Run actions at low volume
below expected threshold
–Increase noise over time
until expected detection
–Plan for actions to turn the
dial up to 11 [w/o breaking all the things]
UNCLASSIFIED 12

13. SOC Exercise: Critical Concepts
• CISO owns risks to execute exercise
– Picks estimated date time to execute
– Give official green-yellow-red for exercise run
– Owns the issues if things go sideways
• Limit knowledge to exercise lifecycle
– Keep “in the know” to least # of individuals
– Space out recon from execution – like most APTs
– Use trusted comms channels for exercise devOps
(i.e. private chat “warroom” for recon, development,
runtime, and reporting creation)
UNCLASSIFIED 13

14. SOC Exercise Major Steps
1. Targeting Selection
2. Determine Expected Outcomes
3. Brief CISO to Signoff
4. Exercise Day
5. Lessons Learned Debrief
UNCLASSIFIED 14

15. Targeting Selection
• State a hypothesis to form a conclusion
“When I run attack A against asset Y, then:
– Defense D should trigger
– Process P should happen
– Response R should occur
• Recon your target’s observables for ops and risks
– Leverage historical logs for events to timeline analysis
– Determine target’s IT & business ops lows and highs
– Research target’s dependencies and if outside TTPs are
dependent on it
UNCLASSIFIED 15

16. Targeting Selection: Crawl-Walk-Run
• Use Crawl-Walk-Run Approach
– Use laser focus by leveraging existing data
– Target improving missional critical processes
– Honor the process to build the trust
• Establishes a track record of success
• Use to obtain immediate results – and use again
to verify lessons learned were implemented
UNCLASSIFIED 16

17. Targeting Selection: Crawl-Walk-Run (cont.)
• Crawl
– Focus on a simple security rule or past IR
– Example: “If I run an exercise actions below, at, and above
a known alerting behavior, what should happen?”
• Walk
– Coordination w/ others outside of immediate security team
– Example: Explicit permission to emulate a particular user
for exercise execution in a mission critical process.
• Run
– Complex exercises once trust is fully established
– Example: Multi week exercise to establish footholds and
pivot to critical datasets
UNCLASSIFIED 17

18. SOC Exercise Selection: Examples
• Detect:
– When will we alert if [insert attack vector]
occurs to [insert system] at this time?
– Has [insert tool] been configured to
operational environment beyond defaults?
– What [insert tool(s)] will record an audit log
of this event’s observables?
UNCLASSIFED 18

19. SOC Exercise Selection: Examples
• Respond:
–Does [insert tool] respond as expected
to [insert attack vector]?
–Does SOC report on response capture
on [insert observables]?
–What [insert tool(s)] will audit log SOC’s
actions to response of the event?
UNCLASSIFED 19

20. SOC Exercise Selection: Examples
• Escalate:
–Did the exercise events escalate as
expected through [insert IR plan]?
–Are we meeting [insert IR plan]
response times?
–When [critical person in IR contact plan]
is out, what happens?
UNCLASSIFED 20

21. Determine Expected Outcomes
1. Develop timeline of major actions
2. Quantify possible risks to execution
3. Establish measurements to actions
UNCLASSIFIED 21

22. Determine Expected Outcomes: Actions
• DO
– Execute attack below know defense thresholds
– Build actions to increase over time to & exceed
known thresholds
– Plan for execution failure failbacks
• DON’T
– Execute an attack you have not tried in a offline test
environment version first
– Always assume an attack will be successful
UNCLASSIFIED 22

23. Determine Expected Outcomes: Risks
• Identify Overall Risks to Ops
– Critical operations at peak use
– Execution during other incidents
– Breaking an unrecoverable system
– Unknown interdependencies
• Quantify Probability & Impacts for Events
UNCLASSIFIED 23
Action Probably Impact Risks
Threshold T triggers for
Observable O1 in SOC tool S1
High Low Recon identified
manufacturer
defaults by
Method M1

24. Determine Expected Outcomes: Measure
• Results of actions
– Should be able to map to STIX observable-
objects (previously CyBOX)
– Should [hopefully] create security events in
your security tooling for actions & reactions
– Be able to query in your SIEM [hopefully?]
UNCLASSIFIED 24
Be aware that SOC may be watching for your
recon as part of its monitoring 

25. Determine Expected Outcomes: Measure
• Use objective measurements – i.e.
– Time between planned actions vs. actual
– Time taken to execute action (start to finish)
– Frequency of action
– Successful/failure of action
• Prepare for *other* findings
– Unusual output to action that was not expected
– Discovering other possible defense weaknesses
– Ensuring documented disclosure post- exercise
UNCLASSIFIED 25

26. Briefing CISO to Signoff
• Be clear on purpose of the exercise
“Evaluating our DLP sensors at X location on our
network will detect PII exfiltration to policy Y”
• Explain why this is important – i.e.
– Validating that Tool X is configured to policy Y
– Testing Response X performs A, B, & C actions
– Verifying Threshold H will trigger under these
conditions.
UNCLASSIFIED 26

27. Briefing CISO to Signoff (cont.)
• Discuss action plan and its risks
– CISO must be able to communicate to their
bosses if something goes wrong
– Confirm accuracy to business & IT ops
expectations from recon observables
– Adjust to CISO’s expectations
• Establish parameters for when to start/stop
– Business impacts
– IT Ops impacts
– Actions exceed expected thresholds
UNCLASSIFIED 27

28. CISO Signoff (cont.)
• Establish exercise agreement in writing
– This is your “get out of jail free card”
– Require CISO to sign physically or digitally
– If physically testing, carry on-person
• Explicitly obtain exercise day(s) w/ CISO
– Expect date to change based on ops rhythms
– Establish method of comms prior to “go-time”
UNCLASSIFIED 28

29. Exercise Day
• Preparation
– Comms channels for WAR online war room
– Team in channel and prepared for run
– CISO gives Red Leader green light in writing
• Runtime
– CISO in channel to watch progress of activity
– Report event action progress in channel
– Immediately report any seen unusual observables
– Explicitly report when TTP thresholds exceeded
before pumping the volume up to 11
UNCLASSIFIED 29

30. Exercise Day: Measure Actions
• Record objective findings for events
– Event outputs
– Changes in timetable for event execution
– Individuals responding to event execution
• Document unexpected outcomes – i.e.
– Attack did not execute successfully
– Attack exposed an additional, out-of-scope
vulnerability
UNCLASSIFIED 30

31. Exercise Day: End of Exercise
• Confirm with CISO on exercise closure
CISO may want opportunity to test an
additional action in scope if undetected
• Hold post-op call with team + CISO
– Collect thoughts on runtime results
– Document any lessons learned during runtime
– Discuss anything that was unexpected
UNCLASSIFIED 31

32. Lessons Learned: Initial Report
• Create briefing on exercise
– Audience – minimum:
• Exercise team + CISO
• Person(s) impacted by exercise testing
• Assume exec level
– Provide purpose of exercise
– Show execution timeline for expected to
actual results
– Keep details on tap for live briefing for Q&A
UNCLASSIFIED 32

33. Lessons Learned: Initial Report
• Briefing – be interactive!
– Maintain blameless
environment
(just the facts!)
– Solicit feedback from
those targeted in
exercise
– Propose possible
solutions to mitigate
or correct for discussion points
UNCLASSIFIED 33

34. Lessons Learned: Post Report
• Provide actionable goals and timelines to
resolve findings
– Leverage input from targets of exercise
– Quantify with results of exercise
• Build goals into performance and risk
management plans
– Project development or maintenance tasks
– Adjusting existing monitoring controls
– Documenting into security plans for auditors
UNCLASSIFIED 34

35. Questions?
UNCLASSIFIED 35