Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Big Data For Threat Detection & Response

139 views

Published on

Slides used at the University of Edinburgh SIGINT group (cybersecurity society). Covering what is big data, the value for security use cases, hunting for threats/actions, using Splunk to detect and respond, SIEM use and some useful searches (which were demoed).

Published in: Data & Analytics
  • Be the first to comment

  • Be the first to like this

Big Data For Threat Detection & Response

  1. 1. BIG DATA FOR THREAT DETECTION & RESPONSE Harry McLaren – Managing Consultant at ECS Sam Farmer – Security Operations Specialist
  2. 2. WHO AM I? HARRY MCLAREN •Alumnus of Edinburgh Napier (Now a Mentor) •Managing Security Consultant at ECS • Big Data Consultancy (Splunk) • Building SOC Technology (SIEM) Copyright © - ECS 2018
  3. 3. •Building/Running Security Operations Centres •Fastest Growing Practice in UK •Supports 80% of Top UK Banks •FTSE 100 Client Base Copyright © - ECS 2018
  4. 4. AGENDA • Introduction & Agenda • Security Operations Overview • Challenge: Monitoring, Detection & Hunting • Solution 1: Big Data, Splunk & Heterogeneous Data • Example: Example of Advanced Threat Activity • Solution 2: SIEM, Platform Evolution & Frameworks • Successful SIEM Deployments & Operation • Splunk User Group & Questions Copyright © - ECS 2018
  5. 5. Copyright © - ECS 2018
  6. 6. ADVANCED THREATS ARE HARD TO FIND • Human directed • Goal-oriented • Dynamic (adjust to changes) • Coordinated • Multiple tools & activities • New evasion techniques • Fusion of people, process, & technology • Contextual and behavioral • Rapid learning and response • Share info & collaborate • Analyze all data for relevance • Leverage IOC & Threat Intel Threat Attack Approach Security Approach Technology People Process Copyright © - ECS & Splunk 2018
  7. 7. ADVANCED THREATS ARE HARD TO FIND • Human directed • Goal-oriented • Dynamic (adjust to changes) • Coordinated • Multiple tools & activities • New evasion techniques Threat Attack Approach Security Approach Technology People Process Analytics-driven Security Connecting Data and People Risk-Based Context and Intelligence Copyright © - ECS & Splunk 2018
  8. 8. ADVANCED THREATS ARE HARD TO FIND ▶ Continuously Protect the business against: • Data Breaches • Malware • Fraud • IP Theft ▶ Comply with audit requirements ▶ Provide enterprise Visibility ▶ 70% to 90% improvement with detection and research of events ▶ 70% to 95% reduction in security incident investigation ▶ 10% to 30% reduction in risks associated with data breaches, fraud and IP theft ▶ 70% to 90% reduction in compliance labor Top Goals Top Splunk Benefits Copyright © - ECS & Splunk 2018
  9. 9. ADVANCED THREATS ARE HARD TO FIND Servers Storage DesktopsEmail Web Transaction Records Network Flows DHCP/ DNS Hypervisor Custom Apps Physical Access Badges Threat Intelligence Mobile CMDB Intrusion Detection Firewall Data Loss Prevention Anti-Malware Vulnerability Scans Traditional Authentication Copyright © - ECS & Splunk 2018
  10. 10. SOLUTION: SPLUNK, THE ENGINE FOR MACHINE DATA Custom Dashboards Report & Analyze Monitor & Alert Developer Platform Ad-hoc Search References – Coded fields, mappings, aliases Dynamic information – Stored in non-traditional formats Environmental context – Human maintained files, documents System/application – Available only using application request Intelligence/analytics – Indicators, anomaly, research, white/blacklist Real-Time Machine Data On-Premises Private Cloud Public Cloud Storage Online Shopping Cart Telecoms Desktops Security Web Services Networks Containers Web Clickstreams RFID Smartphones and Devices Servers Messaging GPS Location Packaged Applications Custom Applications Online Services DatabasesCall Detail Records Energy Meters Firewall Intrusion Prevention Copyright © - ECS & Splunk 2018
  11. 11. EXAMPLE OF ADVANCED THREAT ACTIVITIES .pdf executes & unpacks malware overwriting and running “allowed” programs Threat Intelligence Auth - User Roles Host Activity/Security Network Activity/Security Transaction Gain Access to System Create Additional Environment Conduct Business Svchost.exeCalc.exe Attacker hacks website. Steals .pdf files Web Portal Attacker creates malware, embed in .pdf Read email, open attachment Emails to the target EMAIL HTTP (web) session to command & control server Remote control, Steal data, Persist in company, Rent as botnet WEB Copyright © - ECS & Splunk 2018
  12. 12. EXAMPLE OF ADVANCED THREAT ACTIVITIES .pdf executes & unpacks malware overwriting and running “allowed” programs Threat Intelligence Auth - User Roles Host Activity/Security Network Activity/Security Transaction Gain Access to System Create Additional Environment Conduct Business Svchost.exeCalc.exe Attacker hacks website. Steals .pdf files Web Portal Attacker creates malware, embed in .pdf Read email, open attachment Emails to the target EMAIL HTTP (web) session to command & control server Remote control, Steal data, Persist in company, Rent as botnet WEB Intrusion Detection Credit card transmitted Endpoint Security Hacker tool found Windows Authentication Admin account used Copyright © - ECS & Splunk 2018
  13. 13. CONNECT THE “DATA-DOTS” TO SEE THE WHOLE STORY Persist, Repeat Attacker, know relay/C2 sites, infected sites, IOC, attack/campaign intent and attribution Where they went to, who talked to whom, attack transmitted, abnormal traffic, malware download What process is running (malicious, abnormal, etc.) Process owner, registry mods, attack/malware artifacts, patching level, attack susceptibility Access level, privileged users, likelihood of infection, where they might be in kill chain • Third-party Threat Intel • Open source blacklist • Internal threat intelligence • Firewall • IDS / IPS • Vulnerability scanners • Web Proxy • NetFlow • Network • Endpoint (AV/IPS/FW) • Malware detection • PCLM • DHCP • OS logs • Patching • Active Directory • LDAP • CMDB • Operating System • Database • VPN, AAA, SSO Delivery, Exploit Installation Gain Trusted Access Upgrade (escalate) Lateral Movement Data Gathering Exfiltration Persist, Repeat Threat Intelligence Auth - User Roles Host Activity/Security Network Activity/Security Copyright © - ECS & Splunk 2018
  14. 14. CONNECT THE “DATA-DOTS” TO SEE THE WHOLE STORY phishing Download from infected site 1 2 5 6 7 8 3 4 Threat Intelligence Data Host or ETDR Data Web or Firewall Data Threat Intelligence Data Identity Data Threat Intelligence Auth - User Roles Host Activity/Security Network Activity/Security Delivery Exploitation & Installation Command & Control Accomplish Mission EMAIL WEB EMAIL WEB Copyright © - ECS & Splunk 2018
  15. 15. Security Information & Event Management (SIEM) Software products and services combine security information management (SIM) and security event management (SEM). They provide real-time analysis of security alerts generated by network hardware and applications. Source: Wikipedia & Gartner Copyright © - ECS 2018
  16. 16. SIEM USE CASES Security & Compliance Reporting Real-time Monitoring of Known Threats Detecting Unknown Threats Fraud Detection Insider Threat Incident Investigations & Forensics Copyright © - ECS & Splunk 2018
  17. 17. SIEM EVOLUTION Term Initially Coined in 2005 by Gartner v1.0 Ticketing & Workflow Integrations v1.5 Risk Based Analysis & “Intelligence” v2.0 “Next-Gen SIEM”v3.0 Initial Rule Sets & Event Queues Environment Awareness & Correlation Searches Risk Management & Threat Data Intelligence Machine Learning & Orchestration Copyright © - ECS 2018
  18. 18. SO WHAT'S THE PROBLEM? Copyright © - ECS 2018
  19. 19. SIEM COMPONENT PARTS RULES Correlation Searches, Thresholds & Grouping CONTEXT Organisational Awareness & Impact Assessment FRAMEWORKS Scalable Functionality & User Empowerment INTEGRATION Data Compatibility, Extensibility & Workflow Management Copyright © - ECS 2018
  20. 20. Source: Splunk Developer PortalCopyright © - ECS & Splunk 2018
  21. 21. A B C D INTEGRATION Maximize cross-silo visibility by on-boarding ALL data sources. Automate repetitive tasks and setup orchestration for the rest. PREPARATION Understand your project’s input and output requirements. Champion the project and identify project dependencies. SUCCESS CRITERIA Identify the problem(s) you’re trying to solve. Document the risks/threats and the controls/mitigations. EMBEDDING Position SIEM project as part of transformative change. Enable and engage SecOps to own and evolve platform. SUCCESSFUL SIEM Copyright © - ECS 2018
  22. 22. QUESTIONS?
  23. 23. WHO AM I? SAM FARMER •Alumnus of Edinburgh Napier •Security Operations Specialist at ECS • Security Operations SME • Security Monitoring (SOC) • SIEM Implementation • Threat Hunter Copyright © - ECS 2018
  24. 24. UNICORNS DON’T EXIST
  25. 25. DIAMOND MODEL
  26. 26. BASIC SEARCHING sourcetype=XmlWinEventLog:Microsoft-Windows-Sysmon/Operational |eval length=len(CommandLine) | where length>1000 | table host CommandLine length | sort - length Copyright © - ECS 2018
  27. 27. GROUPING sourcetype=XmlWinEventLog:Microsoft-Windows-Sysmon/Operational | bin span=10m _time | search (process=svchost.exe OR process=lsass.exe OR process=dns.exe OR process=explorer.exe) | stats earliest(_time) as earliest, latest(_time) as latest, values(process) as recon_process, dc(process) as processes, by host | where processes>2 | eval duration=(latest-earliest) Copyright © - ECS 2018
  28. 28. STACKING sourcetype="stream:http" | bin span=1d _time | stats count as curr_count by _time | appendcols [search index=botsv1 sourcetype="stream:http" | stats count as total_count] | eval avg_count = round(total_count/30,0) | stats list(avg_count) as "Average Count", list(total_count) as "Total Count", values(curr_count) as curr_count Copyright © - ECS 2018
  29. 29. STANDARD DEVIATION | bin span=3m _time | stats count as curr_count by _time | streamstats window=1 current=false avg(curr_count) as prev_count | eval growth=curr_count-prev_count | stats avg(curr_count) as average stdev(curr_count) as std_dev latest(curr_count) as latest_vol latest(_time) as lt count(eval(curr_count>150)) as qualifying count as tots | eval conf_int=average+(3.69*(std_dev/sqrt(tots))) | where ((latest_vol>150 AND qualifying=1 AND relative_time(now(), "-4m")<lt) OR (latest_vol>conf_int AND qualifying>=8)) | rename average as "Average" std_dev as "Standard Deviation" latest_vol as "Latest Volume" lt as "Latest Time" qualifying as Qualifying tots as Total conf_int as "Confidence Interval" | convert ctime("Latest Time") timeformat="%H:%M:%S %d/%m/%y" Copyright © - ECS 2018
  30. 30. SPLUNK USER GROUP - EDINBURGH • When: • TBA (Register for Invite) • Where: • Edinburgh Napier University, 10 Colinton Road, Edinburgh, EH10 5DT • Register: https://usergroups.splunk.com/group/spl unk-user-group-edinburgh.html Copyright © - ECS 2018
  31. 31. CONTACT @cyberharibu harry.mclaren@ecs.co.uk harrymclaren.co.uk Copyright © - ECS 2018

×