Apache Metron Profiler in Depth

© Hortonworks Inc. 2011 – 2016. All Rights Reserved1
Apache Metron  
Profiler
Nick Allen, Committer and PMC Member, Apache Metron
Cyber Bootcamp 2017

Agenda
⬢ Introduction
⬢ Anatomy of a Profile
⬢ Creating Profiles
⬢ Data Sketches
⬢ Profiles
⬢ Implementation

Introduction

The Profiler
⬢ A generalized, extensible solution for extracting feature sets from high throughput, streaming data
⬢ Generates a profile describing the behavior of an entity; a host, user, subnet or application
⬢ A foundational component for both security model building and alerting in Metron
Profiler
Models
Alerts

Background: Enrichments
⬢ Enriching telemetry with contextual clues is invaluable
– Dramatically improves threat triage and response
– Another foundational component for security model building (expands the feature set)
⬢ Security telemetry in Metron is ‘sticky’
– Enrichments ‘stick’ as the telemetry progresses through the system
Threat Intel
GeoIP
Asset DB

The Problem
⬢ Enrichments operate within the context of a single message
– Simple, efficient and scalable for most enrichment and triage scenarios
⬢ Insufficient for other scenarios
– Looking across time; trending
– Looking across data sources; correlation
– Looking at aggregate behaviors; How is an “application” or “user” behaving as a whole?

The Profiler
⬢ The Profiler creates logical windows that span both time and data sources
⬢ The user defines a Profile that operates on these logical windows
– A Profile consumes each message within a given window
– Identifies an Entity; the subject of interest; a server, user, subnet, or application name
– Produces a Result; a summary of an Entity’s behavior within the window
t1 t2 t3

Profiles are Time Series
⬢ A Profile executed on a given window results in a unique Result for each Entity
⬢ As a Profile is executed over time, a series of Results starts to emerge
⬢ Rearranging the data slightly, we can see that a unique time series results for each (Profile, Entity) pair

Anatomy of a Profile

“Hello World” Profile
{
"profile": “hello-world",
"foreach": "ip_src_addr",
"init": { "count": "0" },
"update": { "count": "count + 1" },
"result": “count"
}
⬢ A Profile that counts the number of telemetry messages for each IP source
address

{
"init": { "count": "0" },
"result": “count"
}
Name; identifier for the profile
address

{
"init": { "count": "0" },
"result": “count"
}
Entity; a Stellar expression
Maintain a count for each unique source IP address
address

{
"init": { "count": "0" },
"result": “count"
}
Initialize a counter at the start of each window
address

{
"init": { "count": "0" },
"result": “count"
}
Increment the counter for each message in the
window
address

{
"init": { "count": "0" },
"result": “count"
}
window
Return the count at the end of the window
address

{
"init": { "count": "0" },
"result": “count"
}
window
address
0.5 0.5
hello-
world
[10.0.0.3] count = 431
0.5 0.3 0.5
0.5 0.5
[10.0.0.5] count = 372

{
"init": { "count": "0" },
"result": “count"
}
window
address
[Stellar]>>> PROFILE_GET(“hello-world", “10.0.0.3", PROFILE_FIXED(30, "MINUTES"))
[451, 448]
[Stellar]>>> PROFILE_GET(“hello-world", "10.0.0.5", PROFILE_FIXED(30, "MINUTES"))
[234, 176]

Creating Profiles

Profile Debugger
⬢ Creating and refining profiles is an iterative process
⬢ Iterating against a live stream of data is slow, difficult and error-prone
⬢ The Profile Debugger was created to provide a controlled execution
environment to create, refine, and troubleshoot profiles
⬢ The Profile Debugger is executed within the Stellar REPL

Launch the Stellar REPL
[root@node1 0.4.1]# bin/stellar -z node1:2181
Stellar, Go!
Please note that functions are loading lazily in the background and will be
unavailable until loaded fully.
{es.clustername=metron, es.ip=node1, es.port=9300, es.date.format=yyyy.MM.dd.HH}
[Stellar]>>>
[Stellar]>>> %functions PROFILER
PROFILER_APPLY, PROFILER_FLUSH, PROFILER_INIT
⬢ Connecting to Zookeeper (-z node1:2181) is optional, but helpful

Create a Profile Definition
[Stellar]>>> conf := SHELL_EDIT()
[Stellar]>>> conf
{
"profiles": [
{
"profile": "hello-world",
"onlyif": "exists(ip_src_addr)",
"init": { "count": "0" },
"result": "count"
}
]
⬢ Use the editor to create your profile definition(s)

Initialize the Profiler
[Stellar]>>> profiler := PROFILER_INIT(conf)
[Stellar]>>> profiler
Profiler{1 profile(s), 0 messages(s), 0 route(s)}
⬢ Create a Profile execution environment
⬢ Outputs the number of profiles defined, the number of messages applied, and the
number of routes taken.
⬢ A route is defined when a message is applied to a specific profile.
⬢ If a message is applied and not needed by any profile, then there are no routes.
⬢ If a message is needed by one profile, then one route has been defined.
⬢ If a message is needed by two profiles, then two routes have been defined.

Create Telemetry
[Stellar]>>> message := SHELL_EDIT()
[Stellar]>>> message
{
"ip_src_addr": "10.0.0.1",
"protocol": "HTTPS",
"length": "10",
"bytes_in": "234"
}
⬢ Create a message that mimics the type of security telemetry that you expect to profile

Apply Telemetry to the Profiler
[Stellar]>>> PROFILER_APPLY(message, profiler)
⬢ Apply the message to your profile(s) as many times as you like

Flush the Profiler
[Stellar]>>> values := PROFILER_FLUSH(profiler)
[Stellar]>>> values
[{period={duration=900000, period=1669628, start=1502665200000, end=1502666100000},
profile=hello-world, groups=[], value=3, entity=10.0.0.1}]
⬢ A flush is what occurs at the end of each 15 minute period in the Profiler.
⬢ The result is a list of Profile Measurements. Each measurement is a map containing detailed
information about the profile data that has been generated.
⬢ The ‘value’ would have been written to HBase when running in the Profiler topology
⬢ There will always be one measurement for each [profile, entity] pair.
⬢ This profile simply counts the number of messages by IP source address. Notice that the
value is '3' for the entity '10.0.0.1' as we applied 3 messages with an 'ip_src_addr' of
’10.0.0.1'.

Apply “Live” Data
⬢ 'Wash, rinse and repeat' until the profile behaves as you would expect.
⬢ Once you are happy with the profile against a controlled data set, it can
be useful to introduce more complex, live data.
⬢ This example extracts 10 messages of live, enriched telemetry and applies
them to the profile.
[Stellar]>>> %define bootstrap.servers := "node1:6667"
node1:6667
[Stellar]>>> msgs := KAFKA_GET("indexing", 10)
[Stellar]>>> PROFILER_APPLY(msgs, p)

Deploy Your Changes
⬢ It is even simple to deploy your profile directly to the actively running
Profiler topology
⬢ This step requires that you started the REPL with `-z [ZK_URL]`
[Stellar]>>> %functions CONFIG
CONFIG_GET, CONFIG_PUT
[Stellar]>>> CONFIG_PUT("PROFILER", conf)

Data Sketches

Data Sketches
⬢ Data Sketches provide fast, approximate answers to queries about the underlying data
⬢ There are a variety of different types of data sketches, but general characteristics include
⬢ Stream Friendly - Each item of a stream, examined only once, can quickly update a small sketch data structure
⬢ Scalable - Effective for queries that do not scale well; count distinct, quantiles, most frequent items
⬢ Approximate, but with predictable error rates
⬢ Sub-Linear in Size - Required storage space grows more slowly than the input size
⬢ Mergeable (additive) and thus easily support parallelization
⬢ query(sketch(data1 + data2)) == query(sketch(data1) + sketch(data2))
Data
Sketch
Median?
Distinct
count?
Most frequent?

Data Sketches and the Profiler
⬢ The Profiler can persist anything serializable; not just numbers
⬢ Profiler + Data Sketches
⬢ Allows the Profiler to scale to very large data sets
⬢ Allows consumers to ask different queries of the same profile data
⬢ Allows consumers to change the time horizon
⬢ Produce small, lightweight objects that can be stored efficiently

Data Sketches - Example
{
"profile": ”http-length”,
"foreach": ”’global’",
"onlyif": “source.type == ‘bro’ and protocol == 'HTTP'",
"update": { "sk": "STATS_ADD(sk, length)" },
"result": "sk"
}
[Stellar]>>> stats := PROFILE_GET( “http-length", "global", PROFILE_FIXED(24, "HOURS"))
[Stellar]>>> stats
[org.apache.metron.common.math.stats.OnlineStatisticsProvider@79fe4ab9, ...]
⬢ These aren’t just numbers
[Stellar]>>> STATS_MEAN( GET_FIRST( stats))
15979.0625
[Stellar]>>> STATS_PERCENTILE( GET_FIRST(stats), 90)
30310.958
⬢ Ask different queries of the same data ⬢ Merge to change the time horizon
[Stellar]>>> merged := STATS_MERGE( stats)
[Stellar]>>> STATS_PERCENTILE(merged, 90)
29810.992
⬢ A simple Profile that tracks URL length over time

Example Profiles

Failed Logins
{
"profile": ”bad-logins”,
"foreach": ”ip_src_addr",
"onlyif": ”source.type == ‘activedirectory’ and event.type == ‘failed_login’",
“init” : { “count” : “0” }
"update": { “count": ”count + 1" },
"result": “count"
}
⬢ A profile that tracks the number of bad logins by host

Vertex Degree
⬢ View network communication as a directed graph
⬢ The in and out degree can distinguish behaviors
⬢ Anomalies over time can serve as an indicator of compromise
{
"profile": “in-degree",
"onlyif": "source.type == 'yaf'"
"foreach": "ip_dst_addr",
"init": { "in": "HLLP_INIT(5, 6)” }
"update": { "in": "HLLP_ADD(in, ip_src_addr)” }
"result": { “HLLP_CARDINALITY(in)” }
}
client
out >> in
server
in >> out
middleware
in ~= out

Implementation

The Implementation
⬢ A Storm topology that lives outside the critical path
⬢ Profiles are defined in Zookeeper
⬢ Profile Splitter Bolt
– Reads all profile definitions in Zookeeper
– Consumes each message and for each profile
determines…
– Is the message needed by the profile?
– If needed, what is the entity?
– Partitions the data by (Profile, Entity) using a fields
grouping
⬢ Profile Builder Bolt
– Consumes the message, profile definition, and entity
– Updates the state for a (Profile, Entity) pair
– Flushes all state upon receiving a tick tuple resulting in a
Profile Measurement

© Hortonworks Inc. 2011 – 2016. All Rights Reserved
The Implementation
⬢ HBase Bolt
– Persists profile state in HBase
– Generates a row key using a salt, profile name, entity
and time
– Row key needs to be deterministic to allow for retrieval
– The time period is a monotonically increasing number
identifying each period since the epoch
⬢ Kafka Bolt
– Pushes profile data back into Metron
– Allows the user to create alerts based on abnormal
profile values
⬢ Profiler Client
– Based on input parameters, calculates all of the
necessary row keys
– Submits a multi-get using these row keys
– Some row keys may ‘hit’ and others ‘miss’

Q&A
⬢ Questions?
⬢ Join the community
⬢ http://metron.apache.org/
⬢ https://github.com/apache/metron
⬢ More information on the Profiler
⬢ http://metron.apache.org/current-book/metron-analytics/metron-profiler

Apache Metron Profiler in Depth

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Apache Metron Profiler in Depth

Similar to Apache Metron Profiler in Depth (20)

Recently uploaded

Recently uploaded (20)

Apache Metron Profiler in Depth