This document discusses using machine learning for cybersecurity. It outlines some key challenges in cybersecurity like the increasing number of vulnerabilities and novel attacks. It then discusses how machine learning can help by allowing minimum human intervention in prevention, detection, and analysis of attacks. Some implementation challenges of machine learning for cybersecurity are also highlighted, like lack of data and evolving attacks. The document also describes components of a threat prediction platform using machine learning as well as characteristics of security-related data.

1. GVDJ #IDSECCONF2016
machine learning
for cybersecurity

4. USA SOUTH KOREA
NORTH KOREA INDONESIA

5. GVDJ #IDSECCONF2016
security goals
▸ security goals
▸ confidentiality of
information and
resources
▸ integrity of information
and resources
▸ availability of
information and
resources
▸ basic definitions
▸ threat: potential
violation of a security
goal
▸ security: protection
from intentional threats
▸ attack: intentional
violation of a security
goal

6. GVDJ #IDSECCONF2016
security mechanisms
▸ security policies and
mechanisms
▸ policy: statement of what
is and what is not
allowed
▸ mechanism: method or tool
enforcing a security
policy
▸ security is a process, not
a product!
▸ strategies for security
mechanisms
▸ prevention of attacks,
e.g. encryption
▸ detection of attacks,
e.g. virus scanner
▸ analysis of attacks, e.g.
forensic

7. GVDJ #IDSECCONF2016
prevention is a hard task
▸ continuous discovery of vulnerabilities
▸ insecure software and hardware
▸ developers unawareness
goto fail;
goto fail;
goto fail
(february 2014)
heartbleed
(april 2014)
shellshock
(september 2014)

8. GVDJ #IDSECCONF2016
attacks against services
▸ numerous security breaches at popular web services
▸ identities often include real names, addresses, emails,
passwords, etc.
‘;--have i been pwned?
142
pwned websites
1,444,567,928
pwned accounts
39,842
pastes
31,108,929
paste accounts

9. GVDJ #IDSECCONF2016
imbalance of security cycle
▸ increasing imbalance of
security cycle
▸ increasing number of
vulnerabilities
▸ high amount of novel
attacks
▸ high diversity of
malicious software
▸ bottleneck: human
analyst in the loop
▸ manual discovery of
vulnerabilities
▸ manual generation of
attack signatures
▸ manual analysis of
malicious software

10. GVDJ #IDSECCONF2016
conventional detection
▸ conventional attack detection using signatures
▸ ineffective against novel and unknown attacks
▸ inherent delay to availability of novel signatures
▸ analysis obstructed by polymorphism and obfuscation
HEADER APPLICATION PAYLOAD
... IP TCP GET /scripts/ ..%c1%9c.. /system32/cmd.exe
TCP ..%c1%9c.. NIMDA WORM

11. GVDJ #IDSECCONF2016
intelligent defence
▸ construction of intelligent security systems
▸ combining computer security and machine learning
▸ minimum human intervention on prevention, detection,
and analysis
▸ challenge in practice
▸ effectivity, efficiency, and robustness
▸ transparency and controlability

12. machine learning for cybersecurity

14. MACHINE
LEARNING
PREDICTION
PLATFORM
HUMAN
INTUITION

15. attack mitigation issues
supervised unsupervised
rules driven
(limited by experiences and expertise)
high rates undetectable attacks
(false negatives)
delayed response
(between detection and prevention)
statistical driven
(improved detection of new attacks)
substantial investigative efforts
(false positives)
alarm fatigue and distrust
(reversion to supervised method)

16. GVDJ #IDSECCONF2016
implementation challenges
▸ lack of data: limited or no history of previous attacks
(required by supervised learning model)
▸ evolving attacks: attackers constantly change their
behaviours, making current models obsolete
▸ limited resources: relying on security analysts to
investigate the attacks can be costly and time consuming

17. GVDJ #IDSECCONF2016
components
THREAT PREDICTION PLATFORM
MODEL
ANALYSTS
PREDICTIONFEATURE
RAW
DATA
ACTION
EVENTS
MODELLING
CONTEXTUAL
MODELLING

18. GVDJ #IDSECCONF2016
components
▸ big data processing system: quantifying features from raw
data
▸ outlier detection system: learning a descriptive model
using features from unsupervised learning process
▸ feedback mechanism and continuous learning: incorporating
analyst input

19. GVDJ #IDSECCONF2016
data characteristics

20. GVDJ #IDSECCONF2016
data characteristics
0.1 data sources
▸ common sources: networking devices and applications log
▸ router, switch, firewall, ids, ips, and load balancer
devices
▸ web, database, and micro services
▸ frontend and backend applications
▸ delivered in realtime from widely distributed systems

21. GVDJ #IDSECCONF2016
data characteristics
0.2 data dimensions and unique entities
▸ volume of raw data: metrics (GB/TB) or number of lines (≥
tens of millions on a daily basis)
▸ specific to behavioural analytics: IP addresses, users,
sessions, etc.
01010101010101001111010111010101
01010001100010010100010011110110
10100100010010010010001010111101
10100111101101001100011110101011
10101110011010111011011101100111
11100000101001100010000011101101
01100001000000011010111110111011
00111001110001000100010011100100
00111011111011110110100100100110
10001010001110111110001001001001

22. GVDJ #IDSECCONF2016
data characteristics
0.3 malicious activity prevalence
▸ under normal circumstances, malicious activities are
extremely rare (generally ≤ 0.1%)
▸ resulting extreme class imbalance in supervised
learning
▸ increasing the difficulty of detection processes
▸ unknown and/or unreported attacks introduce noise into
data
▸ attack vectors can take a wide variety of shapes

23. GVDJ #IDSECCONF2016
big data analytics
DAILY
WEEKLY
MONTHLY
RAW DATA AGGREGATED DATA
JIM ✖ ✖ ✖
FEATURES
ISNEWUSER?
LASTCHANGEDPASSWORD
LASTIPADDRESS
LASTSESSIONLENGTH
.....
.....
.....
.....
.....
NUMBEROFFAILEDLOGIN
JIM

24. GVDJ #IDSECCONF2016
big data analytics
0.1 behavioural signatures
▸ quantifying signatures (often comprises the series of
attack steps) from raw data
▸ quantitative values can be defined by security analysts
▸ extracting features per-entity and per-time-segment basis

25. GVDJ #IDSECCONF2016
big data analytics
0.2 design requirements
▸ capable of analysing ≥ 10 millions entities in daily
basis
▸ capable of updating and retrieving signatures of active
entities, on demand and/or in realtime

26. GVDJ #IDSECCONF2016
big data analytics
0.3.1 process: activity tracking
▸ absorbing the log stream: identifying the entities and
updating corresponding records
▸ in short temporal window: 30 minutes, 1 hour, 12 hours,
or 24 hours.
▸ focus on efficient retrieval for feature computation

27. GVDJ #IDSECCONF2016
big data analytics
0.3.2 process: activity aggregation
▸ computing behavioural features over an interval of time
▸ retrieving all activity records within given interval
▸ aggregating smaller time unit (minutes, hours, days,
weeks) as the feature demands

28. GVDJ #IDSECCONF2016
algorithm selection

29. GVDJ #IDSECCONF2016
algorithm selection

30. GVDJ #IDSECCONF2016
outlier detection
OUTLIER

31. GVDJ #IDSECCONF2016
outlier detection
▸ matrix decomposition-based
outlier analysis
▸ replicator neural networks
▸ density-based outlier
analysis
▸ score interpretation
▸ transforming score to
probabilities
▸ detection ensembles
MATRIX DECOMPOSITION
REPLICATOR NEURAL NETWORKS

32. GVDJ #IDSECCONF2016
continuous learning
▸ overcomes limited analyst
bandwidth
▸ overcomes weaknesses of
unsupervised learning
▸ actively adapts and
synthesises new models
PREDICTACT
TRAIN

33. GVDJ #IDSECCONF2016
example: open network insight
leveraging insights from flow and packet analysis

34. GVDJ #IDSECCONF2016
example: open network insight
advantages

35. GVDJ #IDSECCONF2016
example: open network insight
how it works

36. GVDJ #IDSECCONF2016
example: entrada
network data analytics platform

37. GVDJ #IDSECCONF2016
summary
▸ current problems of
security
▸ automatisation of
attacks
▸ massive amount of novel
malicious code
▸ defences involving
manual actions (often
ineffective)
▸ machine learning in
security
▸ adaptive defences using
learning algorithms
▸ automatic detection
and analysis of
threats

38. QUESTIONS?