Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Adversarial Simulation Nickerson/Gates Wild West Hacking Fest Oct 2017


Published on

The evolution chain in security testing is fundamentally broken due to a lack of understanding, reduction of scope, and a reliance on vulnerability “whack a mole.” To help break the barriers of the common security program we are going to have to divorce ourselves from the metrics of vulnerability statistics and Pavlovian risk color charts and really get to work on how our security programs perform during a REAL event. To do so, we must create an entirely new set of metrics, tests, procedures, implementations and repeatable process. It is extremely rare that a vulnerability causes a direct risk to an environment, it is usually what the attacker DOES with the access gained that matters. In this talk, we will discuss the way that Internal and external teams have been created to simulate a REAL WORLD attack and work hand in hand with the Defensive teams to measure the environments resistance to the attacks. We will demonstrate attacks, capabilities, TTP’s tracking, trending, positive metrics, hunt integration and most of all we will lay out a road map to STOP this nonsense of Red vs BLUE and realize that we are all on the same team. Sparring and training every day to be ready for the fight when it comes to us. This is an update to our 2016 Brucon talk. We plan to discuss what have we accomplished regarding the above in the last year. We plan to show how we have progressed with the automation of attacker activities and event generation using MITRE’s Cyber Analytics Repository & CAR Exploration Tool (CARET) along with pumping these results to Unfetter ( for aggregation and display in a useful format.

Published in: Technology
  • How will you feel when your Ex girlfriend is in bed with another man? Don't let this happen. Get her back with... ◆◆◆
    Are you sure you want to  Yes  No
    Your message goes here
  • Secrets To Working Online, Hundreds of online opportunites you can profit with today! ★★★
    Are you sure you want to  Yes  No
    Your message goes here
  • link to the video
    Are you sure you want to  Yes  No
    Your message goes here

Adversarial Simulation Nickerson/Gates Wild West Hacking Fest Oct 2017

  1. 1. aka: Some new term to use because we keep screwing up terminology and treating people like children with a crayon box Adversarial Modeling Exercises Simulation
  2. 2. HI…I’m Chris
  3. 3. and…I’m Chris
  4. 4. Chris Gates - Sr. Security Engineer - Uber Twitter: @carnal0wnage Blog: Talks:
  5. 5. Trigger Warnings ● I'm 20yrs in and feel like we are going backwards ● I respect everyone in this room as a peer that can help solve the hard problems ● My opinion is my own, but I bet a few of you share it =) ● I'm SO sick of all the b1tching ● Status quo is unacceptable ● No matter what stupid example I use, its to make it light hearted and not INTENDED to offend ● If I do offend you… oh well ● My finger of blame points with one finger forward but 3 at myself ● If you disagree or wanna add something in SPEAK UP! I don’t bite. ● I don’t have answers, but I am trying to figure it out. ● I’ll work hard to not waste your time
  6. 6.
  7. 7. Problems With Testing Today •  Limited metrics •  Increased Tech debt •  Fracturing of TEAM mentality •  Looks NOTHING like an attack •  Gives limited experience •  Is a step above Vuln Assessment •  Is NOT essential to the success of the organization •  Is REALLY just a glorified internal pentest team
  8. 8.
  9. 9. Building a successful internal Adversarial Simulation Team
  10. 10. Tester Terminology:
 Gotta get a few things straight first We keep screwing up terms •  Vulnerability Assessment person ( U ran a Vuln scanner?) •  Penetration Tester ( U hit go go empire/msf buttons) •  Red Teamer ( Above + went laterally used bloodhound, CME and got DA? MAYBE, got “sensitive stuff”) •  Purple Teamer (U did all of the above but charged more to talk with the defense teams during/after the test) •  ADVERSARIAL ENGINEER (U exist to simulate real world TTP’s, generate experience, and provide metric scoring of corporate readiness /resistance to attack… or just another $$$ Pentester) or some other random shit we will make up next to kill Adv. Eng.
  11. 11. Last Year (Jan 2016) ● There is the MITRE ATT&CK **thing** ○ It’s weird but ok, we should look into it and see if it’s useful ● It makes sense to lay out what our security tools and appliances actually do ● It makes sense to map your detection rules against ATT&CK to see where you have coverage ● Let’s start thinking about automating adversarial simulation
  12. 12. Conduct a attacker capability Assessment
  13. 13. Conduct a defense controls inventory
  14. 14. Wins Stats (mutual) ● Tons of net new findings ● First time we could visualize things we were missing ● Visualize heatmap of coverage for controls ● Proved effectiveness of tools ● Saves Millions in M&A assessments ● Provided defenders a “map of blindspots” magic decoder ring. ● Huge social awareness in org ● Community rally to evolve testing and generating defensive metrics ● Defense is getting sexy ○ Testing prevention/detection to attacks is way more interesting than DA.
  15. 15. Losses stats (mutual) ● Awesome idea … shitty ability to convince others (enrollment) ● Measurement is still a mystery / WIP ○ We need some/more defensive telemetry ● Turned into a pentest team ● So many techniques to cover (150+) ● When people leave, so does the info ( institutionalized) ● Reports are a bitch ○ Non-existent, we have to make new reporting templates ● So much data you lose things in the pile ● Too new for tools to handle/manipulate * livin on edge* ● “errybody” is “developing and aligning” meaning no one is DOING .
  16. 16. So Manual…
  17. 17. Excel HELL
  18. 18. NOPE!
  19. 19. What to tackle next? ● Scalability ● Measurement ● Stale Data ● Data Decay ● Environments change….. A LOT! ● Detection was as good as the person ticket was routed to. ● Consistency ● Tons of manual labor/ time vampire ● Loss of faith (takes forever)
  20. 20. We don’t Scale!
  21. 21. Why don’t we measure anything? ● What does it do? ● Does it work? ● Does it do what it says it does? ● Does it fit? ● Is it consistent? ● Is it accurate? ● Is it reliable? ● Is it repeatable?
  22. 22. Time for a new term. + = Trademark CG & CN
  23. 23. Trademark CG & CN Telemetry is an automated communications process by which measurements and other data are collected at remote or inaccessible points and transmitted to receiving equipment for monitoring.
  24. 24. This Year (2017) So, I challenged work to let me implement/continue of the initiatives from last year
  25. 25. Validate Detection Capabilities Mapped our detection rules to MITRE ATT&CK ○ It was valuable to see where we had coverage and where we didn’t with our various tools Refactored all rules to map to their ATT&CK Phase and Technique
  26. 26. Validate Detection Capabilities Refactored all rules to be version controlled (yaml). ○ Making easier for all analysts to contribute. ○ Utilize various yaml fields for tool enrichment
  27. 27. (Automated) Rule QA Detecting threats requires logging pipelines o  Activity logs from various systems o  Logs must be transported to something that aggregates and processes §  Each step in this process is a place the pipeline could fail §  Pipelines can go down for a variety of reasons ●  Vendor schema changes ●  Vendor outages ●  Log processing issues (malformed logs / amount of logs) ●  Firewall (host/network) changes ●  OS issues ●  Exceeding licenses ●  Forgetting to pay the bill o  How do we ensure our rules are still current? Effective? AND working?
  28. 28. Complex Log Pipelines
  29. 29. (Automated) Rule QA Wanted a way to programmatically validate the detection pipeline is functioning o  No/Some alerts doesn’t automatically mean all is well o  It probably means a log pipeline is down o  Daily tests of the pipeline seem like a good idea §  Give us metrics around time of action → phantom case Wanted a way to perform ongoing rule unit testing (QA) o  If a detection rule has zero means: §  We aren’t pwned or... §  The rule doesn’t work correctly (when was the last time it did?) o  For every detection rule, we want to have a way to generate an event that causes the rule to fire §  Some rules fire all time, others almost they work? o  Doing this manually is no fun/not scalable (bat file) :-(
  30. 30. (Automated) Rule QA How ●  Piggyback off the new Rule Framework and alert monitoring system (Phantom Cyber) ●  Use Virtualization (vagrant) so we can reset the host as required o  Ex. Executing malware or programs that leave host in an undesirable state ●  Use a Task Queue (Celery) to handle all the execution / requests / responses ●  Flask endpoint to receive results from Phantom Cyber
  31. 31. (Automated) Rule QA
  32. 32. (Automated) Rule QA Use the YAML fields to hold our data
  33. 33. (Automated) Rule QA
  34. 34. (Automated) Rule QA
  35. 35. Where to get rules ● ● ● ● :-) Full’s a PITA. There isn’t a great resource for these rules, you have to make them yourself.
  36. 36. Adversarial Simulation ● What do you do when you have a couple hundred “attacker” actions and an engine to queue them up? ● You automate them to do attacker stuff :-) ● ●  SOON...working thru open sourcing process :-/
  37. 37. Adversarial Simulation ●  Broken down by MITRE ATT&CK Phases and APT Groups Examples:
  38. 38. Adversarial SimulationSingle Technique
  39. 39. Adversarial SimulationScenarios
  40. 40. Adversarial SimulationScenarios
  41. 41. Adversarial SimulationScenarios + Basic Reporting
  42. 42. Adversarial SimulationOther Cool Projects Atomic Testing by Red Canary
  43. 43. Tools DumpsterFire
  44. 44. Tools Blue Team Training Toolkit
  45. 45. Unfetter Using unfetter to track progress/coverage ● ● Mapping attack groups ● Assessments
  46. 46. Unfetter Mapping attacker groups
  47. 47. Unfetter Mapping attacker groups
  48. 48. Unfetter Mapping attacker groups
  49. 49. Unfetter Assessments
  50. 50. Unfetter Assessments
  51. 51. Unfetter Assessments
  52. 52. Metta Maps of the Future Possible Plugins ●  Unfetter ( grab, post, report) ●  CarbonBlack ●  Splunk ●  Phantom ●  Anything with an API is possible C2 ○  Terraform/ansible/etc to build c2 endpoints Malicious file attachments ○ Malicious office macros ○ ○ ○ AWS ○  Programmatically build vuln AWS services/infra - can BT detect?
  53. 53. Additional Telemetry ● Total Coverage ● Mean Time to Detection ● Mean Time to Remediation ● % Successful Eradication ● Protection Metrics ● Automated vs Manual Protection ● Automated vs Manual Detection ● Automated vs Manual Response ● Defender proficiency & capability ● Tool/Product Reliance ● Product coverage heatmaps ● Pre-Flight verification ● Net fluctuations
  54. 54. Defensive Coverage Attacker/Defender Difficulty METTA Defender Capability Self Healing Defense Offensive Inputs Response Auto scaling Protection Capability
  55. 55. Chris Gates Twitter: @carnal0wnage Blog: Talks: Video: GitHub: Chris Nickerson Twitter: @indi303 Talks: Video: Company: