Doing advanced analytics in streaming architectures presents unique challenges around the tradeoff of having more context vs. performance. Typically performance and scalability requirements mandate that each message in a stream be operated on without the context of other messages in the stream that may have come before. In this talk, we will talk about using sketching algorithms to engineering a compromise which allows us to consider historical state without compromising scalability. What we found analyzing the capabilities of many similar SIEMs and cybersecurity platforms is that a good portion of the advanced anaytics boil down to either simple rules enriched with the ability to do statistical baselining, set existence, and set cardinality computations. These operations are necessarily difficult to do in-stream, so often they're done after the fact. We look at ways to open up these analytics to stream computation without sacrificing scalability. Specifically, we will introduce the infrastructure built for Apache Metron to perform these kinds of tasks. We will cover the novel integration between an Apache Storm and Apache Hbase and orchestrated by a custom domain specific language called Stellar to take all the sting out of constructing sketches and using them to accomplish simple and more advanced analytics such as statistical outlier analysis in stream. CASEY STELLA, Principal Software Engineer, Hortonworks

1. 1 © Hortonworks Inc. 2011–2018. All rights reserved
Just the Sketch
Advanced Streaming Analytics
Casey Stella
Principal Software Engineer & VP Apache Metron

2. 2 © Hortonworks Inc. 2011–2018. All rights reserved
Apache Metron: A Cybersecurity Analytics Solution
• Metron provides a scalable advanced security analytics framework to offer a centralized
tool for security monitoring and analysis
• Ultimately, this means that we provide a solution to ingest, enrich and detect anomalies in
disparate data sources
• Metron was initiated at Cisco in 2014 as OpenSOC
• Metron was submitted to the Apache Incubator in December 2015
• Metron graduated to a top level project in April 2017

3. 3 © Hortonworks Inc. 2011–2018. All rights reserved
Apache Metron: The Stack
• We aggressively use the Hadoop stack for both batch as well as streaming processing of
data.
• We use
• Apache Zookeeper for distributed configuration management
• Apache HBase for random access key/value data
• Apache Storm as a stream processing framework
• HDFS for long-term storage of processed data
• Apache Solr and Elasticsearch for low latency querying
• We’ve built a UI to display these alerts to security analysts and allow them to manage
them.

4. 4 © Hortonworks Inc. 2011–2018. All rights reserved

5. 5 © Hortonworks Inc. 2011–2018. All rights reserved

6. 6 © Hortonworks Inc. 2011–2018. All rights reserved

7. 7 © Hortonworks Inc. 2011–2018. All rights reserved
Enrichment by any means necessary
• There are a couple of different ways to enrich, each with their own semantics
Changing Scope
Retrieval
Semantics
Ingestion
Semantics
Metron
Solution
Static/Slow-moving Event Key/Value Lookup Batch HBase Enrichment
Static/Slow-moving Event Complex Batch Summarizer
Dynamic Event Key/Value Lookup Streaming
Streaming HBase
Enrichment
Static/Slow-moving Event Complex Batch Model as a Service
Dynamic Multi-event Complex Streaming Profiler

8. 8 © Hortonworks Inc. 2011–2018. All rights reserved
The Profiler
t = 1 t = 2 t = 3 t = n
The Profiler creates logical windows
that span both time and data sources
 Trending across time
 Anomaly detection across time
 Stores summaries and sketches in
Hbase for efficient retrieval in-
stream
 Provides a query language that
allows seasonal adjustment
Approx. Data
Sketch
Approx. Data
Sketch
Approx. Data
Sketch
Approx. Data
Sketch
Combined
Baseline
Statistic

9. 9 © Hortonworks Inc. 2011–2018. All rights reserved
The Profiler: Data Access Semantics
• Provide a mechanism to store user-defined summaries for every k minutes
• Provide a mechanism to query stored profile data
• Fixed Lookback – Look back a fixed amount of time
• Seasonal Adjusted Window – Look back for a time period applying seasonal adjustment
• Often the data we are interacting with behaves seasonally and queries should support
seasonal adjustment
• from 1 hour ago
• from 1 hour ago until 30 minutes ago
• 1 hour window every 24 hours from 56 days ago including this day of
the week excluding holidays:us, weekends

10. 10 © Hortonworks Inc. 2011–2018. All rights reserved
Data Sketches
• Data sketches provide fast, approximate answers to common questions about data
• Statistical questions (e.g. median, percentile, standard deviation)
• Set Operations (e.g. containment, existence, cardinality)
• Sampling
• They are (generally) sub-linear in size because they are approximate
• They are able to be merged and questions able to be asked of the merged results
• query(sketch(data1) + sketch(data2)) = query(sketch(data1 + data2))

11. 11 © Hortonworks Inc. 2011–2018. All rights reserved
Data Sketches and the Profiler
• The Profiler can persist anything; not just numbers
• We can, for instance, persist a sketch per time period
• Allows the profiler to scale to large datasets
• Allows questions to be asked of data across long time ranges
• Allows time ranges to be specified at read-time rather than write-time
• Allows users to ask different questions of the same data

12. 12 © Hortonworks Inc. 2011–2018. All rights reserved© Hortonworks Inc. 2011 – 2016. All Rights Reserved31
Data Sketches - Example
{
"profile": ”http-length”,
"foreach": ”’global’",
"onlyif": “source.type == ‘bro’ and protocol == 'HTTP'",
"update": { "sk": "STATS_ADD(sk, length)" },
"result": "sk"
}
[Stellar]>>> stats := PROFILE_GET( “http-length", "global", PROFILE_FIXED(24, "HOURS"))
[Stellar]>>> stats
[org.apache.metron.common.math.stats.OnlineStatisticsProvider@79fe4ab9, ...]
⬢ These aren’t just numbers
[Stellar]>>> STATS_MEAN( GET_FIRST( stats))
15979.0625
[Stellar]>>> STATS_PERCENTILE( GET_FIRST(stats), 90)
30310.958
⬢ Ask different queries of the same data ⬢ Merge to change the time horizon
[Stellar]>>> merged := STATS_MERGE( stats)
[Stellar]>>> STATS_PERCENTILE(merged, 90)
29810.992
⬢ A simple Profile that tracks URL length over time

13. 13 © Hortonworks Inc. 2011–2018. All rights reserved
Where in the world is Carmen San Diego?: Anatomy of a Solution
• One way to find anomalous behavior on a network is to monitor the locations from
which a user logs in. If a user logs in from a vastly different place than usual, this could
be circumstantial evidence of malicious behavior.
• The trick to any analytic is how to define “vastly different” and “than usual”
• We can track the distance from the geographic center of the user’s previous login events over time
• We can compare that to the distribution of the distances from the geographic center for all users to
decide if it’s truly abnormal.
• In order to do this, we’ll need to
• Ingest authentication data that associates users with login events with a Parser
• Track the location of the logins across users in a scalable way using the Profiler
• Enrich login events by interrogating the user’s login history and determining if their login is
sufficiently abnormal to bump their threat level

14. 14 © Hortonworks Inc. 2011–2018. All rights reserved
Ingest the auth data
{ "parserClassName" : "org.apache.metron.parsers.csv.CSVParser"
,"sensorTopic" : "auth"
,"parserConfig" : {
"columns" : { "user" : 0,"ip" : 1, "timestamp" : 2}
}
,"fieldTransformations" : [
{
"transformation" : "STELLAR"
,"output" : [ "hash" ]
,"config" : {
"hash" : "GEOHASH_FROM_LOC(GEO_GET(ip))"
}
}
]}
Auth data has 3 fields
We’ll add a new field
called “hash” that is
the geohash of the ip

15. 15 © Hortonworks Inc. 2011–2018. All rights reserved
Profiler: Track user login locations
{
"profile": "locations_by_user",
"foreach": "user",
"onlyif": "exists(hash) && hash != null && LENGTH(hash) > 0",
"init" : {
"s": "MULTISET_INIT()"
},
"update": {
"s": "MULTISET_ADD(s, hash)"
},
"result": "s“
}
This profile tracks user
login behavior
We will compute a
profile for each user
We’ll use the multiset
Stellar functions to
track the geohash and
the # of occurrences

16. 16 © Hortonworks Inc. 2011–2018. All rights reserved
Enrich the auth data with more context
"enrichment": {
"fieldMap": {
"stellar" : {
"config" : [
"geo_locations := MULTISET_MERGE( PROFILE_GET( 'locations_by_user', user,
PROFILE_FIXED( 4, 'HOURS')))",
"geo_centroid := GEOHASH_CENTROID(geo_locations)",
"geo_distance := TO_INTEGER(GEOHASH_DIST(geo_centroid, hash))",
"geo_locations := null"
]
}
}
,"fieldToTypeMap": { }
}
Get the set of geohashes (and
occurrences) for the user over the
last 4 hours
Calculate the
geographic center of
the logins
Find the distance of the
current login from the
geographic center and
create a new field
“geo_distance” to hold
that for every login
event

17. 17 © Hortonworks Inc. 2011–2018. All rights reserved
Profiler: Baseline across all users
{
"profile": "geo_distribution_from_centroid",
"foreach": "'global'",
"onlyif": "exists(geo_distance) && geo_distance != null",
"init" : {
"s": "STATS_INIT()"
},
"update": {
"s": "STATS_ADD(s, geo_distance)"
},
"result": "s"
}
Track the distribution
of distances from the
geographic center over
the previous 4 hours
For all users
We’ll use the STATS
Stellar functions to
track the distribution
of our newly enriched
field, geo_distance

18. 18 © Hortonworks Inc. 2011–2018. All rights reserved
Compute the threat given global context and per-user context
"threatIntel": {
"fieldMap": {
"stellar" : {
"config" : [
"geo_distance_distr:= STATS_MERGE( PROFILE_GET( 'geo_distribution_from_centroid',
'global', PROFILE_FIXED( 4, ’HOURS')))",
"dist_median := STATS_PERCENTILE(geo_distance_distr, 50.0)",
"dist_sd := STATS_SD(geo_distance_distr)",
"geo_outlier := ABS(dist_median - geo_distance) >= 5*dist_sd",
"is_alert := exists(is_alert) && is_alert",
"is_alert := is_alert || (geo_outlier != null && geo_outlier == true)",
"geo_distance_distr := null"
]
}
}
Get the statistical distribution of the
‘geo_distance’ field for all users
Decide if the
geo_distance is an
outlier by testing how
many standard
deviations it is from
the median
Update “is_alert” accordingly. If this
is true, then we will need to triage
the alert level.

19. 19 © Hortonworks Inc. 2011–2018. All rights reserved
Triage Threat
"triageConfig" : {
"riskLevelRules" : [
{
"name" : "Geographic Outlier",
"comment" : "Determine if the user's geographic distance from the centroid of the
historic logins is an outlier as compared to all users.",
"rule" : "geo_outlier != null && geo_outlier",
"score" : 10,
"reason" : "FORMAT('user %s has a distance (%d) from the centroid of their last login
is 5 std deviations (%f) from the median (%f)', user, geo_distance, dist_sd, dist_median)"
}
],
"aggregator" : "MAX"
}
Because this is only a
circumstantial
indicator, we’ll only
give this a threat score
of 10
We’ll need to ensure
the security analyst has
enough context to
make a decision here.
In a normal system,
there would be many
rules triaging the
threat, the maximum
score would be taken
to the score for the
message.

20. 20 © Hortonworks Inc. 2011–2018. All rights reserved
Questions?

21. 21 © Hortonworks Inc. 2011–2018. All rights reserved
Thank you
Come visit us at the Hortonworks Booth and attend the Cybersecurity Birds of a Feather on Wednesday!