The document discusses the challenges faced by operations, IT, and security engineers in managing and analyzing machine-generated logs, emphasizing the limitations of traditional SIEM systems and the need for machine learning solutions. It explores behavioral analysis and various machine learning techniques like device fingerprinting and browser fingerprinting to automate data analysis and improve detection capabilities in cybersecurity. The document concludes with a call for incorporating machine learning to effectively adapt to evolving environments and handle complex data patterns.

1. Rise of the machines

2. 2
Agenda
• Some pains of Ops./IT engineers and Security folk
• Rise of the machines: what’s behavioral analysis and what’s
Machine Learning
• (Example) Device fingerprinting from 802.11 traffic
• (Example) Browser fingerprinting from HTTP headers
• Wrap up and resources

3. 3
Typical stuff Operations/IT/Security engineers
do to be in control
• Have important information logged to logging service.
• Perhaps have some Security Information and Event Management
(SIEM) solution(s).
• If you want online awareness, you implement and deploy event
monitors and setup dashboards.
• If “something interesting” or “something important” happens, you
want to understand and respond such that you keep the business
running, fix what you can and wake up people to fix what you can’t.
• So we/they/someone-else writes rules (x happened y times, foo
happened with some relationship to bar…)

4. 4
Problems with the approach
• Services and websites change over time:
• Deployments (infrastructure change, design change, functionality
change…)
• Routine or occasional maintenance to frontend/backend
• A/B testing
• Security tests
• Etc.

5. 5
Problems with the approach
• Traffic changes:
• Seasonality: hourly, daily, weekly, holidays, etc.
• Rare events: successful campaigns, bad reviews
Changes aren’t necessarily an unexpected anomaly
or an attack. So, how to distinguish?

6. 6
Problems with the approach
Rules written in SIEM systems almost always either
• over-generalize
or
• under-generalize
So they are too simplistic to capture complex reality
or over-fitted such that they don’t generalize well
• After a while are too complicated to maintain
• What happens when need to change a rule?
• What about managing ordering and dependencies?
• How to debug?

7. 7
Problems with the approach
• Too many logs
• Machine generated but humanly handled
• Too many signals to monitor and decide on simultaneously

8. 8
Problems with the approach
Too few skilled people to handle. How to find them?
Are there both security analysts and data scientists?

9. 9
Let the machines handle machine generated logs
So, we want to use Machine Learning in order to
automate adaptation to change and to be able to
handle volumes constantly and repeatedly.

10. 10
3 hottest terms lately
• Cyber (Security)
• Big Data
• Machine Learning
Now… Imagine all three together…
You can probably think of a few names from only last 3 or so months
Raising millions of $$ when saying:
big-data//machine- learning//cyber-security

11. 11
What's Behavioral?
When we use the term Behavioral we mean that we're looking at
attributes that are not (necessarily) related to the content but rather
to information that describe or that may describe the behavioral
properties of the actual content.
Behavioral analysis focuses on the observable relationship of behavior
to the environment.
So, instead of addressing particular properties of communication
content and context we use information about how that
communication takes place and being used. For example, we look at
timing, at methods used, etc.

12. 12
What's Behavioral?
• Some say
Behavioral Analysis
and actually mean
Anomaly Detection
• Others say
Behavioral Analysis
and actually mean
Machine Learning

13. 13
What’s Machine Learning?
• You don’t develop rules, instead you develop software that
discovers the rules by itself.
• You sometimes don’t even design what input (features) to feed to
the learning algorithms, as those can be (sometimes) learned too
• You sometimes don’t even need to implement the feature
extraction, as such code can be (sometimes) be auto-written too
Try reading A Few Useful Things to Know About Machine Learning by Pedro Domingos (CACM, Volume 55, Issue 10, October 2012, pp. 78-
87) – you may also want to read the great commentary about this paper here and here

14. 14
When we say Machine Learning we mean
that…
• our solution needs not be explicitly programmed or configured in
order to be well adapted and tuned to particular installation, setup,
environment, changes in the application, changes in traffic, etc.
• we'd like our solution to work out of the box without need of
human guidance or intervention. Instead, we'd like it work and to
adapt to changes by using examples instead of explicitly being
programmed.
• we hope to automate some (hopefully, most or all) of the domain
expertise network analyst work by learning from examples.
• hoping to be as good at the task as human experts, but scale better.

15. 15
Machine Learning
Machine learning systems automatically learn programs from data.
This is often a very attractive alternative to manually constructing
them.

16. 16
Some facts about Machine Learning
• Learning/training:
• You try to fit a function or a family of functions from your input (think
that your input is ultimately a series of k-tuples)
• Applying:
• You feed a new k-tuple and get a result

17. 17
But how does the model building process
actually work?
All machine learning algorithms (the ones that build the models)
basically consist of the following three things:
• A set of possible models to look thorough
• A way to test whether a model is good
• A clever way to find a really good model with only a few test

18. 18
Ways to classify machine learning algorithms
Supervised vs. Unsupervised
Classification (vs. clustering) vs. Regression
Online vs. Offline: (Streaming (learn and apply as you go) vs. Iterations (down to
only 1))
All input is there vs. Missing/incomplete data
.
.
.

19. 19
Surprising facts about Machine Learning
• Although there are many off-the-shelf tools to help doing machine
learning it is almost always harder to do it right
• on real world problems,
• on real customer data,
• constantly,
• in scale and
• in quality
• Roughly 95% or more of the efforts are due to data collection and
preparation (missing values, correctness, relevance, representation,
balancing, cleaning, …)

20. 20
Surprising facts about Machine Learning
• Many-times simpler common-sense algorithms outperform (quality,
scale, maintainability, …) complicated algorithms when big-data is
available
• It not so much about what algorithm you use but how much and quality
of the data you have
• Data representation many-times matter more (Deep Learning)
• Ensembles of simple-specialized algorithms usually do better than one
monolithic complex algorithm (Ensemble, Arbitration, …)

21. 21
We want learning: implicit, automatic, dynamic
• Not need to write rules, not manage them
• Want the rules to be learned automatically and
• Don’t want to set/change thresholds
• Want the thresholds to be determined automatically and
dynamically
• I want to know about rare events that matter without needing to
define what rare means and without needing to define important
• Become better as more data becomes available

22. 22
Example: 802.11 device fingerprinting
An empirical study of passive 802.11 Device Fingerprinting
Christoph Neumann, Olivier Heen, Stéphane Onno
Proceedings of 32nd International Conference on Distributed
Computing Systems Workshops (ICDCSW 2012), Workshop on
Network Forensics, Security and Privacy (NFSP'12)

23. 23
802.11 device fingerprinting
• 802.11 device fingerprinting is the action of characterizing a target
device through its wireless traffic.
• This results in a signature that may be used for identification,
network monitoring or intrusion detection.

24. 24
802.11 device fingerprinting
• The fingerprinting method is passive by just observing the traffic
sent by the target device.
• Focus on network parameters which can be easily extracted using
standard wireless cards
• Method should work also for encrypted 802.11 traffic
• Method should not be detected by attackers hard to cheat with
adversarial traffic
• Accurate

25. 25
802.11 device fingerprinting
Many passive fingerprinting methods rely on the observation of one
particular network feature, such as the rate switching behavior or the
transmission pattern of probe requests.
In this work, the researchers evaluated a set of global wireless
network parameters with respect to their ability to identify 802.11
devices.
They restricted themselves to parameters that can be observed
passively using a standard wireless card.
Used information extracted by Radiotap or Prism headers

26. 26
802.11 device fingerprinting
Machine Learning? Show me the ML!
• Features: Network parameters – observable features
• Transmission rate [Mbit/ µsec] – different card vendors and models
have variations
• Frame size [bytes] – differences in broadcast frame sizes implicitly
identify wireless devices
• Medium access time [µsec] – time since medium is idle and until device
starts sending its own frame
• Transmission time [µsec] – frame duration -- time it takes to send a
frame (approximate by frame size divided by transmission rate)
• Frame inter-arrival time [µsec] – time from end (start) of one frame and
end (start) of next frame on the same direction

27. 27
802.11 device fingerprinting
Machine Learning? Show me the ML!
Computed features:
• Foreach frame type (data frames, probe requests, …)
• Foreach sender over the medium
• Maintain frequency histograms per observable feature
• Periodically,
• transform frequency histograms to proportional histograms

28. 28
802.11 device fingerprinting
Machine Learning? Show me the ML!
Researchers in the paper used a supervised learning approach.
Learn:
Assume fixed/known set of devices
Characterize devices using features
Apply:
Compare histograms with learned histograms and MAC addresses.
When a conflict observed vs learned baselines – Alert!

29. 29
802.11 device fingerprinting
How to compare histograms?
Researchers used Cosine similarity.
[Wikipedia]

30. 30
Distance – alternatives?
Depends on what you want to capture? Examples:
• Minkowski distance
• Kullback-Liebler divergence

31. 31
Distance – More alternatives
• Triangular discrimination
• Jensen Shannon
There are many more distance, divergence and similarity measures –
what to use? It depends…

32. 32
Alternative learning?
Instead of supervised learning (requires sterile learning time, needs
examples, rigid, …) let’s do unsupervised:
• Cluster observed histograms by distances
• Assign MAC addresses to clusters
• Look into clusters with more than one MAC address
• If False Positives – be more sensitive to precision or look into
divergences that better capture differences/similarities
• Robust, flexible, assumes very little

33. 33
But what makes a fingerprint?
• Check computed features by type?
• Create one big histogram?
• Create also histograms of inter-dependencies? (Cross product…)
• Hash histograms into something else? (What? How?)
What’s stable? What’s accurate?
It depends on your data, on your representation, on your algorithm…

34. 34
From PHY to L7
An exercise in browser fingerprinting

35. 35
Fingerprinting my browser? Really?

36. 36
Whoa! How?
Mike Sconzo and Brian Wylie have reproducible research which they
presented on ShmooCon 2014
http://nbviewer.ipython.org/github/ClickSecurity/data_hacking/blob/
master/browser_fingerprinting/browser_fingerprinting.ipynb
You can learn methodology of how to do this from HTTP headers in a
scientific manner: Data Scientific manner

37. 37
OK. So, what does this has to do with
Rise of the Machines?
We can now automatically,
• Collect data
• Analyze data
• Organize data
• Insights and conclusions
Find what’s interesting – automatically.
TADA!

38. 38
Usages?
• SPAM detection
• Fraud detection
• Malware detection
• Intrusion detection
• Abuse detection
• Fingerprinting (duh!)
• DDoS detection
• Network traffic classification
• … … … <your idea here>

39. 39
Links and references
• An empirical study of passive 802.11 Device Fingerprinting // Christoph
Neumann, Olivier Heen, Stéphane Onno // Proceedings of 32nd
International Conference on Distributed Computing Systems Workshops
(ICDCSW 2012), Workshop on Network Forensics, Security and Privacy
(NFSP'12)
• https://panopticlick.eff.org/
• ShmooCon 2014: Practical Applications of Data Science in Detection
// Mike Sconzo and Brian Wylie //
http://www.youtube.com/watch?v=8lF5rBmKhWk [start at 35:36]
• http://nbviewer.ipython.org/github/ClickSecurity/data_hacking/blo
b/master/browser_fingerprinting/browser_fingerprinting.ipynb
• Wikipedia {just look for terms}

40. 40
What would I want to learn?
Theory?
Data Science
Statistics
Machine Learning
Statistical Inference
Predictive Analytics

41. 41
What would I want to learn?
Tools?
R (S?) // Octave (Matlab?) // Julia // Haskell // Perl // Python // …

42. 42
How would I learn?
Resources?
• Google is your friend (youtube too)
• Coursera // Udacity // Iversity // EdX // … excellent online courses
• Meetups
• Good old
reading books // university courses // reading academic
papers

43. 43
Thanks to…
Eran Goldstein -- http://www.linkedin.com/in/erangoldstein
Maydan Weinreb -- http://www.linkedin.com/in/maydanw

44. 44
Thank you!
http://il.linkedin.com/in/shlomoyona
s.yona@f5.com
https://www.facebook.com/shlomo.yona