Introducing Analytics, Anomaly Engines and Machine learning concepts in threat detection technologies.

1. #SACON
Detecting Threats with
Analytics and Machine
Learning (ML)
Shomiron DAS GUPTA (GCIA)
Founder, CEO
NETMONASTERY Inc.

2. #SACON
Agenda
■ Dissecting detection systems
■ Why do we need “analytics”
■ Learning systems
■ Anomaly / Heuristics / Dictionaries
■ Machine Learning Use Cases
■ Why ML works / fails

3. #SACON
Dissecting Detection Systems
■ Signature based
■ Anomaly engines
■ Analytics workbench
■ Learning systems

4. Why do we need “Analytics”

5. #SACON
The Real Need for Analytics
■ Cyber security refresh rate
■ Custom payloads from attackers
■ Servers not the target
■ Speed with volume

6. #SACON
Learning Systems
■ Heuristic learning
■ Anomaly engines
■ Spot / Baseline / Profilers
■ Time series analytics
■ Classifiers
■ Unassisted learning

7. #SACON
Heuristics
■ Virus detection, OS Rootkits
v3
Day 14
v2
Day 6
v1
Day 1
v4
Day 42

8. #SACON
Anomaly
■ DDoS Detection, Protocol Obfuscation, Malformed
Data Streams, Application Breach
Fixed Anomaly
Model Structure
Could be traffic behavior,
protocol behavior,
application behavior.
Realtime Data

9. #SACON
Spot / Baseline / Profilers
■ Unordered Action - new rule, new device, long dead
user, database user event
LEARN PHASE EVAL PHASE
Build Model
Transcode model with
feature aggregation
performed on realtime
data flows
Data Data
Evaluation
Identification of
outliers based on
pre approved
model

10. #SACON
Time Series Analytics
■ DDoS, Flow Outliers, protocol breach, zombies
THRESHOLDING DYNAMIC THRESHOLDING
Fixed limits are set to
detect breach in activity
Moving window analysis of
time series data

11. #SACON
Classifiers
■ SPAM, Botnets, Authentication Anomalies
Clustering Process
- Suitable feature selection (PCA)
- Training set (static / dynamic)
- Cleaning training data
- Regression to find mean
- Operations
- Feedback and Re-tuning

12. #SACON
Unassisted Learning
■ SPAM, DNS Detection, L2 Attacks
Alert
Feedback
AnalystSelf Adjusting Loop
Data
Profiler
Model

13. #SACON
When is ML working
■ Credible / Clean training data
■ Positive and timely feedback
■ Picking the right features
■ Consistent feature variation
■ Consistent data pattern

14. #SACON
Where does ML work
■ DNS based detection
■ DDoS / Traffic anomaly
■ SPAM Mail filters
■ Authentication
■ Application modelling
■ Threat Intelligence

15. #SACON
ML is failing
■ Variance challenge
■ The “stale dataset” problem
■ Mass labelling
■ Complex selection challenges

16. #SACON
■ Programming in R / Python
■ Data platforms - Splunk, DNIF
■ Infrastructures - Generic Hadoop, Hortonworks
https://dnif.it
Get started with 100Gb free every month forever
Getting Started with ML

17. Shomiron DAS GUPTA
shomiron@netmonastery.com
+91 9820336050
Thank You!