Improving Network IntrusionImproving Network Intrusion
Detection with Traffic DenoiseDetection with Traffic Denoise
Miroslav Stampar (Croatian Government CERT)Miroslav Stampar (Croatian Government CERT)

IntroductionIntroduction

Network Intrusion-Detection Systems (NIDS) generate
lots of false alerts, false alarms, false positives, etc.

Result of thinking that it is better to catch both
suspicious and harmless events than to miss
something

Lots of irrelevant and/or “frustrating” noise

Security managers/technicians are loosing interest
very fast (serious problem)

Detection/noise problemDetection/noise problem

Basis of a NIDS is “detection”

Like metal detector, NIDS doesn't stop network traffic

It highlights potential threats so that the security
guards, like firewalls, can be more effective

Too much noise results with lessened usability and
benefit

As a side effect, increased management cost

Most of all, lack of focus on real adversaries

Unusable dashboardsUnusable dashboards

Large quantity of eventsLarge quantity of events

Real adversaries vs “junk”Real adversaries vs “junk”

What if we could distinguish regular noise from real
adversaries?

Regular noise should be fairly common on Internet

Subjects targeting multiple ranges should not be
considered as big threat as those targeting only us

Various subjects (i.e. “lurkers”) scan 0.0.0.0/0
searching for easy to exploit systems

Unsophistication at its best

Free (skiddie level) vulnerability assessment

How to capture noise?How to capture noise?

It should be visible by different Internet parties
compared to adversaries targeting only our IP range(s)

If unwanted (inbound) traffic is visible from different IP
ranges in predefined time-span (e.g. 1 day) we can
presume that it is a noise

Two ways:
1) Utilizing multiple (public third-party) IP blacklists
2) Constructing own sensor network made of “silent”
boxes

IP blacklists (generally)IP blacklists (generally)

Multiple third-parties publicly share relatively up-to-
date IP blacklists (i.e. IP addresses with “bad
reputation”)

Inbound traffic coming from such IPs could be safely
DROP-ed (in case of active boundary defense – e.g.
IPS)

AlienVault, CINS, Deepviz, VoIPBL, Turris, etc.

IDSes, honeypots, server logs, “threat intel”, etc.

SSH, HTTP, Telnet, VoIP, NTP, DNS, etc.

IP blacklists (examples)IP blacklists (examples)

IP blacklists (distribution)IP blacklists (distribution)

IPnoise (general)IPnoise (general)

https://github.com/stamparm/ipnoise

Experimental run during the 2016

Setup of 4 sensors on different IP ranges:
37.228.134.XXX (gertrude), 78.47.177.XXX (hans),
5.135.71.XXX (pierre), 216.155.145.XXX (tommy)

Silent *nix boxes capturing (inbound) IP traffic

ICMP and TCP/UDP (ports 1-65535)

type, src_ip, dst_ip, dst_port,
first_timestamp, last_timestamp, count

IPnoise (sample raw data)IPnoise (sample raw data)

IPnoise (statistics)IPnoise (statistics)

Denoise (methodology)Denoise (methodology)

Suricata (ELK) IDS on /20 network (4096 IPs)

Fine tuned ruleset based on ETPro (removed all
“paranoid” rules, rules based on “reputation”,
INFO/POLICY/CORPORATE, etc.) with highly adapted
thresholds

Experimental data based on May 2017 events

Denoise results were obtained by manual processing
(for presentation purposes)

Presenting concept (no final product, yet :)

Events (May 2017)Events (May 2017)

Threats (May 2017)Threats (May 2017)

ResultsResults

On average 35% less events

On average 37% less threats (i.e. (src_ip, event))

ConclusionConclusion

Ignoring, DROPing or lowering the severity of events
for noisy IPs should help to focus on real adversaries

Testing results look promising

Although currently only a concept, personal wish is
that this presentation will give somebody an idea to
further evolve it

If anything, collection of traffic on silent boxes around
the 0.0.0.0/0 could pinpoint the wandering (noisy) IP
addresses (more than 1000 per day)

Appendix (ipsum)Appendix (ipsum)

https://github.com/stamparm/ipsum

Collecting IP addresses from >30
different online blacklists

Creation of different (ip)sets based
on minimum number of occurrences

For example, 3.txt contains all IPs
found on at least 3 blacklists

Daily automatic updates

Questions?Questions?