Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Art into Science 2017 - Investigation Theory: A Cognitive Approach

758 views

Published on

This presentation was delivered at Art into Science 2017 in Austin, TX. I discuss the ongoing cognitive crisis in information security, and present original research methods and results related to the investigation process.

Published in: Technology
  • Be the first to comment

Art into Science 2017 - Investigation Theory: A Cognitive Approach

  1. 1. Investigation Theory A Cognitive Approach Chris Sanders
  2. 2. Chris Sanders (@chrissanders88)  Analyst @ FireEye  Founder @ Rural Tech Fund  PhD Researcher  GSE # 64  BBQ Pit Master  Author:  Practical Packet Analysis  Applied NSM  Investigation Theory Course
  3. 3. Symptoms of a Cognitive Crisis 1. Demand for expertise greatly outweights supply 2. Most information cannot be trusted or validated 3. Inability to mobilize and tackle big systemic issues
  4. 4. Ethnography of the SOC “An analyst’s job is highly dynamic and requires dealing with constantly evolving threats. Doing the job is more art than science. Ad hoc, on-the-job training for new analysts is the norm." Sundaramurthy, S. C., McHugh, J., Ou, X., Rajagopalan, S. R., & Wesch, M. (2014). An anthropological approach to studying CSIRTs. Network, 100, 2.
  5. 5. Ethnography of the SOC “The profession [security] is so nascent that the how-tos have not been fully realized even by the people who have the knowledge…the process required to connect the dots is unclear even to analysts. Sundaramurthy, S. C., McHugh, J., Ou, X., Rajagopalan, S. R., & Wesch, M. (2014). An anthropological approach to studying CSIRTs. Network, 100, 2.
  6. 6. Symptoms of a Cognitive Crisis 1. Demand for expertise greatly outweights supply 2. Most information cannot be trusted or validated 3. Inability to mobilize and tackle big systemic issues
  7. 7. The Cognitive Revolution 1. Understand the processes used to draw conclusions 2. Develop repeatable methods and techniques 3. Build and advocate training that teaches practitioners how to think
  8. 8. What separates novice and expert analysts?
  9. 9. Mapping the Investigation  Sample:  Novice and expert analysts  Methodology:  30+ case studies  Stimulated recall interviews  Focus on individual investigations of varying types  Perform key phrase analysis – analyze results
  10. 10. Key Phrase Mapping  Dual Process Theory  Intuition: Implicit, unconscious, fast  Reflection: Explicit, controlled, slow Intuition Experimentation Restructuring Imagination Incubation Metacognition Evaluation Goal Setting Making Plans Reflection Analytically Viewing Data Rule-Based Reasoning Considering Alternatives
  11. 11. Results Novices Experts Intuition Metacognition Reflection
  12. 12. Analyzing the Flow of the Investigation
  13. 13. Investigations as Mental Labyrinths  The investigation is the core construct of information security.  How do we study them when everyone has a different toolset?  Follow the Data! Alert OSINT Reputation File Hash Sandbox Behaviors AV Detections (VT) Imphash More File Hashes Friendly Host Network PCAP Host Windows Logs Security Log System Log App LogRegistry File System Hostile Host Network PCAP Flow
  14. 14. Studying the Investigation Process
  15. 15. Studying the Investigation Process
  16. 16. What data did analysts look at first? 72% 16% 12% Observed PCAP Flow OSINT Data Suggests:  Analysts prefer a higher context data set…  …even if other data sets are available  …even if lower context data sets can lead to a resolution.
  17. 17. Did the first move affect analysis speed? Data Suggests:  While PCAP provides richer context, it may slow down the investigation if that’s where you start  Starting with a lower context data source can increase speed when working with higher context data 16 10 9 PCAP Flow OSINT Avg Time to Close
  18. 18. What happens when Bro data replaces PCAP? 46% 25% 29% Observed (Bro) Bro Flow OSINT 72% 16% 12% Observed (PCAP) PCAP Flow OSINT
  19. 19. What happens when Bro data replaces PCAP? 16 10 9 PCAP Flow OSINT Avg Time to Close (PCAP) 10 10 11 Bro Flow OSINT Avg Time to Close (Bro) Data Suggests:  Better organization of high context data sources can yield improvements in analysts performance
  20. 20. What data sources were viewed most and least frequently? Data Suggests:  Network data is used more frequently than host data…  …even when host data can be used exclusively to resolve.  …even when easy access is provided to host sources.  Revisting data is more prevalent on higher context data sources Data Sources Viewed Data Sources Revisited PCA P 84% Flow 11% OSIN T 5%
  21. 21. How many steps were taken to make a disposition judgement? Data Suggests:  At some point, the number of data sources you investigate impacts the speed of the investigation  Understanding where data exists and when to use it can impact analysis speed 6 12 9 3 0 5 10 15 6-10 11-15 16-20 21-25 Number of Steps 9 12 14 24 0 5 10 15 20 25 30 6-10 11-15 16-20 21-25 Avg Time to Close
  22. 22. Did analysts investigate friendly or hostile systems first? 9% 91% Observed Friendly Hostile Data Suggests:  Analysts are more compelled to investigate unknown external threats than internal systems  Analysts don’t fully understand their own techniques 41% 59% Friendly Friendly Hostile
  23. 23. Thank You! Mail: chris@chrissanders.org Twitter: @chrissanders88 Blog: chrissanders.org Training: chrissanders.org/training

×