SlideShare a Scribd company logo
1 of 38
Collecting
Useful Security
Data for
Incident
Response
NOT BIG DATA, ANY DATA
Martin Holste
Halo effects of a good incident response
process:
¡ Fight FUD with truth
¡ IT ops love truth, therefore ops loves security,
therefore security gets access to data/
resources
¡ When security=visibility and security!=NO,
good things happen
WHY ARE WE DOING IR?
Context
Artifacts
Events
Staff
Political Will
REQUIREMENTS FOR IR
Tools/data
¡ IR isn’t done until you’ve answered “why”
¡ Why=hard
¡ Need to fully understand threat motives via
events and artifacts
¡ This requires quickly answering tough
questions, which means searching,
summarizing, and drilling down
CONTEXT IS THE GOAL
¡ Search, summarize, drill-down on events and
artifacts
¡ Leverages both local and global data
¡ Parallax of heterogeneous data sources
describing the same action
ELSA PROVIDES CONTEXT
Parallax: derive
depth from
different points
of view
EVENT PARALLAX
1.1.1.1
<->
2.2.2.2
Firewall
• 200000 bytes
downloaded
Proxy
• GET example.com/
evil.exe
• category=uncategorized
Bro
• MD5 deadbeefdeadbeef
• GET example.com/
evil.exe
• Country code=RU IDS
• Packed executable
downloaded
Windows
• User is admin
CONTEXT VIA TIMELINE
9/12 23:00 81.x.x.x GET company.com/directory.html
9/13 08:00 Email from admin@throwaway.com
9/13 09:05 Firewall connection 82.x.x.x
9/13 09:05 GET request to evil.com 82.x.x.x
9/13 09:05 Bro file MD5 from evil.com 82.x.x.x
9/13 09:05 IDS Packed Executable 82.x.x.x
9/14 07:00 IDS outbound RAR 83.x.x.x
Recon
Exploit
Exfil
Phish
ELSA searches to construct the timeline:
1.  Find exploit:
sig_msg:packed groupby:dstip |
subsearch(!
class:url groupby:site,dstip) |
subsearch(!
category:uncategorized!
groupby:srcip,dstip)!
BUILDING THE TIMELINE
This gives us the attacker IP 82.x.x.x, victim IP
1.1.1.1. Continue with victim-centric searches.
2.  Find exfil: 1.1.1.1 groupby:sig_msg!
3.  Find bait: 1.1.1.1 groupby:email |
subsearch(class:bro_smtp
groupby:subject)!
4.  Find recon: site:company.com
uri:directory.html!
BUILDING THE TIMELINE
1.1.1.1 groupby:dstip |
subsearch(class:url,srcip)!
Results from 1.1.1.1 groupby:dstip:
Subsearch evaluates:
class:url srcip:2.2.2.2 OR srcip:
3.3.3.3!
SUBSEARCHES
Groupby Count
2.2.2.2 5
3.3.3.3 1
InternalsEXTENDING ELSA
Parse • Syslog-NG
Normalize • elsa.pl
Forward • ELSA plugins
Archive • MySQL
Index • Sphinx (indexer)
INDEX DATA FLOW
Parse • ELSA
QueryParser.pm
Query • Sphinx (searchd)
Get Rows • MySQL
Transform • ELSA plugins
Connect • ELSA plugins
SEARCH DATA FLOW
ParsersEXTENDING ELSA
¡ Parsers are XML that go in /etc/elsa/
patterns.d/
¡ ELSA install.sh will merge all files there into
merged.xml, which is what syslog-ng uses
¡ Documentation online at ELSA’s Google Code
project page
WRITING PARSERS
Syslog (RFC 3164) message format:
<priority|facility> <timestamp> <program>:
<message>
<13> Sep 14 01:02:03 crond[1234]: Started!
WRITING PARSERS
Example:
<ruleset id=“…”>
<pattern>program</pattern>
<rules>
<rule id=“…” class=“integer class ID”>
<patterns>
<pattern>matching code</pattern>
WRITING PARSERS
Parser types:
¡ ESTRING: Slurp until given char or chars
¡ QSTRING: Slurp between chars
¡ NUMBER: Slurp digits
¡ The rest are not really needed
WRITING PARSERS
@<parser type>:<fieldname>:<chars>@!
Parse: http://example.com/ with:!
http://@ESTRING:site_name:/@!
Extracts “example.com” to field “site_name”
http:/@QSTRING:site_name:/@!
Same but slower
PARSER COMPONENT STRUCTURE
Always try to use ESTRING
http://example.com/!
What if scheme is “https” or “ftp?”
@ESTRING:://@@ESTRING:site_name:/@!
First ESTRING moves pointer to after the first
double slashes it finds, but does not extract a
field name.
PARSER STRATEGIES
If you don’t have a character to start the parsing
evaluation, you may get unexpected results,
e.g.:
my firewall log URL: http://example.com/!
This matches as well, which may not be desired
behavior.
You can work around this with
<pattern>program</pattern> in the
<ruleset> element.
ESTRING GOTCHAS
Field names are stored in an abstracted format:
Names map to field_order in SQL schema,
class_id dependent
INTEGRATING PARSERS WITH ELSA
Integer name i0 i1 i2 i3 i4 i5
Field_order 5 6 7 8 9 10
String name s0 s1 s2 s3 s4 s5
Field_order 11 12 13 14 15 16
fields_classes_map
(class_id, field_id,
field_order)
classes
(id, class)
fields
(id, field,
field_type)
SQL SCHEMA
<rule>
…
<examples>
<example>
<test_message program=“prog”>test text
<test_values>
<test_value name=“s0”>expected val
!
pdbtool test merged.xml!
TESTING PARSERS
PluginsEXTENDING ELSA
ELSADatasource
Info
Connector
Transform Export
Stats
Post
Processor
Forwarder
PLUGINS
Goal: Provide ELSA access to an HR database to
augment queries
Desired ELSA query:
datasource:hr department:finance
groupby:username!
Finds all users in finance!
EXAMPLE: HR DATABASE PLUGIN
Step 1: Define the SQL query on the HRIS:
SELECT user, department FROM users!
Step 2: Add to elsa_web.conf:
“datasources”: {!
“database”: {!
“hr”: {!
“dsn”: …!
“username”: …!
“password”: …!
EXAMPLE: HR DATABASE PLUGIN
..continued:
“fields”: [!
{ “name”: “user” },!
{ “name”: “department” }!
],!
“query_template”: “SELECT %s FROM
users WHERE %s %s”!
}!
EXAMPLE: HR DATABASE PLUGIN
The query template will be built based on the
ELSA query parameters, e.g.
ELSA query:
user:bob!
!
Becomes SQL:
WHERE user LIKE “%bob%”!
EXAMPLE: HR DATABASE PLUGIN
Let’s use it!
datasource:hr department:finance
groupby:user | !
subsearch(class:vpn
groupby:srcip,user) |!
whois | sum(description)!
EXAMPLE: HR DATABASE PLUGIN
Users in finance
department
Correlate with
VPN logins by user
Add srcip
description
Summarize by
description
Yields:
EXAMPLE: HR DATABASE PLUGIN
Description Count
Comcast Corporation 234
Starbucks, Inc. 10
University of Lagos 3What!?! à
Quantcast provides the top one million most visited
sites. It would be nice to know when downloads
occur from sites not on that list.
Step 1: Grab the data: Top 1 million at
https://www.quantcast.com/top-sites
Rank Site
1 google.com
2 youtube.com
3 facebook.com
4 msn.com
5 twitter.com
6 wordpress.com
7 amazon.com
8 ebay.com
9 yahoo.com
10 yelp.com
EXAMPLE: QUANTCAST TRANSFORM
Step 2: Load the data into MySQL
CREATE TABLE quantcast (count INT
UNSIGNED, site VARCHAR(255));!
LOAD DATA LOCAL INFILE
“Quantcast.txt”!
INTO TABLE quantcast IGNORE 6
LINES;!
EXAMPLE: QUANTCAST TRANSFORM
Step 3: Configure the transform
{!
“transforms”: {!
“database”: {!
“quantcast”: {!
! (dsn/user/pass/fields!
query_template)!
EXAMPLE: QUANTCAST TRANSFORM
Step 4: Profit! Find downloads from uncommon
sites (rank greater than ten thousand):!
md5 class:bro_file groupby:site |!
quantcast | sum(count) | has(10000)!
EXAMPLE: QUANTCAST TRANSFORM
There are many places you can add
customization to ELSA. Some plugin ideas:
¡ Connector for ticketing system
¡ LDAP info
¡ Encrypted export
¡ Transform to check if user is logged in
¡ Transform to launch AR Drone attack
THIS IS JUST A START!
MORE INFORMATION
Blog ossectools.blogspot.com
Project Site enterprise-log-search-and-
archive.googlecode.com
Mailing List enterprise-log-search-and-
archive.googlegroups.com
Twitter @mcholste

More Related Content

What's hot

Building Beautiful REST APIs with ASP.NET Core
Building Beautiful REST APIs with ASP.NET CoreBuilding Beautiful REST APIs with ASP.NET Core
Building Beautiful REST APIs with ASP.NET CoreStormpath
 
Workshop: Learning Elasticsearch
Workshop: Learning ElasticsearchWorkshop: Learning Elasticsearch
Workshop: Learning ElasticsearchAnurag Patel
 
Analyse your SEO Data with R and Kibana
Analyse your SEO Data with R and KibanaAnalyse your SEO Data with R and Kibana
Analyse your SEO Data with R and KibanaVincent Terrasi
 
Battle of the Giants - Apache Solr vs. Elasticsearch (ApacheCon)
Battle of the Giants - Apache Solr vs. Elasticsearch (ApacheCon)Battle of the Giants - Apache Solr vs. Elasticsearch (ApacheCon)
Battle of the Giants - Apache Solr vs. Elasticsearch (ApacheCon)Sematext Group, Inc.
 
The ultimate guide for Elasticsearch plugins
The ultimate guide for Elasticsearch pluginsThe ultimate guide for Elasticsearch plugins
The ultimate guide for Elasticsearch pluginsItamar
 
Solr and Elasticsearch, a performance study
Solr and Elasticsearch, a performance studySolr and Elasticsearch, a performance study
Solr and Elasticsearch, a performance studyCharlie Hull
 
Hunting for the secrets in a cloud forest
Hunting for the secrets in a cloud forestHunting for the secrets in a cloud forest
Hunting for the secrets in a cloud forestSecuRing
 
Designing a beautiful REST json api
Designing a beautiful REST json apiDesigning a beautiful REST json api
Designing a beautiful REST json api0x07de
 
From Zero to Production Hero: Log Analysis with Elasticsearch (from Velocity ...
From Zero to Production Hero: Log Analysis with Elasticsearch (from Velocity ...From Zero to Production Hero: Log Analysis with Elasticsearch (from Velocity ...
From Zero to Production Hero: Log Analysis with Elasticsearch (from Velocity ...Sematext Group, Inc.
 
How to Do a Performance Audit of Your .NET Website
How to Do a Performance Audit of Your .NET WebsiteHow to Do a Performance Audit of Your .NET Website
How to Do a Performance Audit of Your .NET WebsiteDNN
 
quick intro to elastic search
quick intro to elastic search quick intro to elastic search
quick intro to elastic search medcl
 
Real-time data analysis using ELK
Real-time data analysis using ELKReal-time data analysis using ELK
Real-time data analysis using ELKJettro Coenradie
 
Elasticsearch - Devoxx France 2012 - English version
Elasticsearch - Devoxx France 2012 - English versionElasticsearch - Devoxx France 2012 - English version
Elasticsearch - Devoxx France 2012 - English versionDavid Pilato
 
Battle of the Giants round 2
Battle of the Giants round 2Battle of the Giants round 2
Battle of the Giants round 2Rafał Kuć
 
2014 spark with elastic search
2014   spark with elastic search2014   spark with elastic search
2014 spark with elastic searchHenry Saputra
 
Apache Solr/Lucene Internals by Anatoliy Sokolenko
Apache Solr/Lucene Internals  by Anatoliy SokolenkoApache Solr/Lucene Internals  by Anatoliy Sokolenko
Apache Solr/Lucene Internals by Anatoliy SokolenkoProvectus
 
Understanding the state of your web application using Apache Kafka, Spark
Understanding the state of your web application using Apache Kafka, SparkUnderstanding the state of your web application using Apache Kafka, Spark
Understanding the state of your web application using Apache Kafka, SparkExist
 
Drupal 8 + Elasticsearch + Docker
Drupal 8 + Elasticsearch + DockerDrupal 8 + Elasticsearch + Docker
Drupal 8 + Elasticsearch + DockerRoald Umandal
 
Elasticsearch, Logstash, Kibana. Cool search, analytics, data mining and more...
Elasticsearch, Logstash, Kibana. Cool search, analytics, data mining and more...Elasticsearch, Logstash, Kibana. Cool search, analytics, data mining and more...
Elasticsearch, Logstash, Kibana. Cool search, analytics, data mining and more...Oleksiy Panchenko
 

What's hot (20)

Building Beautiful REST APIs with ASP.NET Core
Building Beautiful REST APIs with ASP.NET CoreBuilding Beautiful REST APIs with ASP.NET Core
Building Beautiful REST APIs with ASP.NET Core
 
Workshop: Learning Elasticsearch
Workshop: Learning ElasticsearchWorkshop: Learning Elasticsearch
Workshop: Learning Elasticsearch
 
Analyse your SEO Data with R and Kibana
Analyse your SEO Data with R and KibanaAnalyse your SEO Data with R and Kibana
Analyse your SEO Data with R and Kibana
 
The tale of 100 cve's
The tale of 100 cve'sThe tale of 100 cve's
The tale of 100 cve's
 
Battle of the Giants - Apache Solr vs. Elasticsearch (ApacheCon)
Battle of the Giants - Apache Solr vs. Elasticsearch (ApacheCon)Battle of the Giants - Apache Solr vs. Elasticsearch (ApacheCon)
Battle of the Giants - Apache Solr vs. Elasticsearch (ApacheCon)
 
The ultimate guide for Elasticsearch plugins
The ultimate guide for Elasticsearch pluginsThe ultimate guide for Elasticsearch plugins
The ultimate guide for Elasticsearch plugins
 
Solr and Elasticsearch, a performance study
Solr and Elasticsearch, a performance studySolr and Elasticsearch, a performance study
Solr and Elasticsearch, a performance study
 
Hunting for the secrets in a cloud forest
Hunting for the secrets in a cloud forestHunting for the secrets in a cloud forest
Hunting for the secrets in a cloud forest
 
Designing a beautiful REST json api
Designing a beautiful REST json apiDesigning a beautiful REST json api
Designing a beautiful REST json api
 
From Zero to Production Hero: Log Analysis with Elasticsearch (from Velocity ...
From Zero to Production Hero: Log Analysis with Elasticsearch (from Velocity ...From Zero to Production Hero: Log Analysis with Elasticsearch (from Velocity ...
From Zero to Production Hero: Log Analysis with Elasticsearch (from Velocity ...
 
How to Do a Performance Audit of Your .NET Website
How to Do a Performance Audit of Your .NET WebsiteHow to Do a Performance Audit of Your .NET Website
How to Do a Performance Audit of Your .NET Website
 
quick intro to elastic search
quick intro to elastic search quick intro to elastic search
quick intro to elastic search
 
Real-time data analysis using ELK
Real-time data analysis using ELKReal-time data analysis using ELK
Real-time data analysis using ELK
 
Elasticsearch - Devoxx France 2012 - English version
Elasticsearch - Devoxx France 2012 - English versionElasticsearch - Devoxx France 2012 - English version
Elasticsearch - Devoxx France 2012 - English version
 
Battle of the Giants round 2
Battle of the Giants round 2Battle of the Giants round 2
Battle of the Giants round 2
 
2014 spark with elastic search
2014   spark with elastic search2014   spark with elastic search
2014 spark with elastic search
 
Apache Solr/Lucene Internals by Anatoliy Sokolenko
Apache Solr/Lucene Internals  by Anatoliy SokolenkoApache Solr/Lucene Internals  by Anatoliy Sokolenko
Apache Solr/Lucene Internals by Anatoliy Sokolenko
 
Understanding the state of your web application using Apache Kafka, Spark
Understanding the state of your web application using Apache Kafka, SparkUnderstanding the state of your web application using Apache Kafka, Spark
Understanding the state of your web application using Apache Kafka, Spark
 
Drupal 8 + Elasticsearch + Docker
Drupal 8 + Elasticsearch + DockerDrupal 8 + Elasticsearch + Docker
Drupal 8 + Elasticsearch + Docker
 
Elasticsearch, Logstash, Kibana. Cool search, analytics, data mining and more...
Elasticsearch, Logstash, Kibana. Cool search, analytics, data mining and more...Elasticsearch, Logstash, Kibana. Cool search, analytics, data mining and more...
Elasticsearch, Logstash, Kibana. Cool search, analytics, data mining and more...
 

Similar to Not Big Data, AnyData

SplunkLive! - Getting started with Splunk
SplunkLive! - Getting started with SplunkSplunkLive! - Getting started with Splunk
SplunkLive! - Getting started with SplunkSplunk
 
Spark DataFrames: Simple and Fast Analytics on Structured Data at Spark Summi...
Spark DataFrames: Simple and Fast Analytics on Structured Data at Spark Summi...Spark DataFrames: Simple and Fast Analytics on Structured Data at Spark Summi...
Spark DataFrames: Simple and Fast Analytics on Structured Data at Spark Summi...Databricks
 
Getting started with Splunk - Break out Session
Getting started with Splunk - Break out SessionGetting started with Splunk - Break out Session
Getting started with Splunk - Break out SessionGeorg Knon
 
Getting started with Splunk
Getting started with SplunkGetting started with Splunk
Getting started with SplunkSplunk
 
Scout xss csrf_security_presentation_chicago
Scout xss csrf_security_presentation_chicagoScout xss csrf_security_presentation_chicago
Scout xss csrf_security_presentation_chicagoknaddison
 
Sumo Logic Cert Jam - Security & Compliance
Sumo Logic Cert Jam - Security & ComplianceSumo Logic Cert Jam - Security & Compliance
Sumo Logic Cert Jam - Security & ComplianceSumo Logic
 
Develop High Performance Windows 8 Application with HTML5 and JavaScriptHigh ...
Develop High Performance Windows 8 Application with HTML5 and JavaScriptHigh ...Develop High Performance Windows 8 Application with HTML5 and JavaScriptHigh ...
Develop High Performance Windows 8 Application with HTML5 and JavaScriptHigh ...Doris Chen
 
Mdst 3559-03-03-sql-php-2
Mdst 3559-03-03-sql-php-2Mdst 3559-03-03-sql-php-2
Mdst 3559-03-03-sql-php-2Rafael Alvarado
 
Php My Sql Security 2007
Php My Sql Security 2007Php My Sql Security 2007
Php My Sql Security 2007Aung Khant
 
IaaS with ARM templates for Azure
IaaS with ARM templates for AzureIaaS with ARM templates for Azure
IaaS with ARM templates for AzureChristoffer Noring
 
Haystack 2018 - Algorithmic Extraction of Keywords Concepts and Vocabularies
Haystack 2018 - Algorithmic Extraction of Keywords Concepts and VocabulariesHaystack 2018 - Algorithmic Extraction of Keywords Concepts and Vocabularies
Haystack 2018 - Algorithmic Extraction of Keywords Concepts and VocabulariesMax Irwin
 
Rapid Prototyping with Solr
Rapid Prototyping with SolrRapid Prototyping with Solr
Rapid Prototyping with SolrErik Hatcher
 
Information Gathering With Google
Information Gathering With GoogleInformation Gathering With Google
Information Gathering With GoogleZero Science Lab
 
Information Gathering with Google (c0c0n - India)
Information Gathering with Google (c0c0n - India)Information Gathering with Google (c0c0n - India)
Information Gathering with Google (c0c0n - India)Maximiliano Soler
 
Stop the noise! - Introduction to the JSON:API specification in Drupal
Stop the noise! - Introduction to the JSON:API specification in DrupalStop the noise! - Introduction to the JSON:API specification in Drupal
Stop the noise! - Introduction to the JSON:API specification in DrupalBjörn Brala
 
Fire-fighting java big data problems
Fire-fighting java big data problemsFire-fighting java big data problems
Fire-fighting java big data problemsgrepalex
 
Rapid Prototyping with Solr
Rapid Prototyping with SolrRapid Prototyping with Solr
Rapid Prototyping with SolrErik Hatcher
 
Customer Intelligence: Using the ELK Stack to Analyze ForgeRock OpenAM Audit ...
Customer Intelligence: Using the ELK Stack to Analyze ForgeRock OpenAM Audit ...Customer Intelligence: Using the ELK Stack to Analyze ForgeRock OpenAM Audit ...
Customer Intelligence: Using the ELK Stack to Analyze ForgeRock OpenAM Audit ...ForgeRock
 
Ldap Synchronization Connector @ 2011.RMLL
Ldap Synchronization Connector @ 2011.RMLLLdap Synchronization Connector @ 2011.RMLL
Ldap Synchronization Connector @ 2011.RMLLsbahloul
 
OData: Universal Data Solvent or Clunky Enterprise Goo? (GlueCon 2015)
OData: Universal Data Solvent or Clunky Enterprise Goo? (GlueCon 2015)OData: Universal Data Solvent or Clunky Enterprise Goo? (GlueCon 2015)
OData: Universal Data Solvent or Clunky Enterprise Goo? (GlueCon 2015)Pat Patterson
 

Similar to Not Big Data, AnyData (20)

SplunkLive! - Getting started with Splunk
SplunkLive! - Getting started with SplunkSplunkLive! - Getting started with Splunk
SplunkLive! - Getting started with Splunk
 
Spark DataFrames: Simple and Fast Analytics on Structured Data at Spark Summi...
Spark DataFrames: Simple and Fast Analytics on Structured Data at Spark Summi...Spark DataFrames: Simple and Fast Analytics on Structured Data at Spark Summi...
Spark DataFrames: Simple and Fast Analytics on Structured Data at Spark Summi...
 
Getting started with Splunk - Break out Session
Getting started with Splunk - Break out SessionGetting started with Splunk - Break out Session
Getting started with Splunk - Break out Session
 
Getting started with Splunk
Getting started with SplunkGetting started with Splunk
Getting started with Splunk
 
Scout xss csrf_security_presentation_chicago
Scout xss csrf_security_presentation_chicagoScout xss csrf_security_presentation_chicago
Scout xss csrf_security_presentation_chicago
 
Sumo Logic Cert Jam - Security & Compliance
Sumo Logic Cert Jam - Security & ComplianceSumo Logic Cert Jam - Security & Compliance
Sumo Logic Cert Jam - Security & Compliance
 
Develop High Performance Windows 8 Application with HTML5 and JavaScriptHigh ...
Develop High Performance Windows 8 Application with HTML5 and JavaScriptHigh ...Develop High Performance Windows 8 Application with HTML5 and JavaScriptHigh ...
Develop High Performance Windows 8 Application with HTML5 and JavaScriptHigh ...
 
Mdst 3559-03-03-sql-php-2
Mdst 3559-03-03-sql-php-2Mdst 3559-03-03-sql-php-2
Mdst 3559-03-03-sql-php-2
 
Php My Sql Security 2007
Php My Sql Security 2007Php My Sql Security 2007
Php My Sql Security 2007
 
IaaS with ARM templates for Azure
IaaS with ARM templates for AzureIaaS with ARM templates for Azure
IaaS with ARM templates for Azure
 
Haystack 2018 - Algorithmic Extraction of Keywords Concepts and Vocabularies
Haystack 2018 - Algorithmic Extraction of Keywords Concepts and VocabulariesHaystack 2018 - Algorithmic Extraction of Keywords Concepts and Vocabularies
Haystack 2018 - Algorithmic Extraction of Keywords Concepts and Vocabularies
 
Rapid Prototyping with Solr
Rapid Prototyping with SolrRapid Prototyping with Solr
Rapid Prototyping with Solr
 
Information Gathering With Google
Information Gathering With GoogleInformation Gathering With Google
Information Gathering With Google
 
Information Gathering with Google (c0c0n - India)
Information Gathering with Google (c0c0n - India)Information Gathering with Google (c0c0n - India)
Information Gathering with Google (c0c0n - India)
 
Stop the noise! - Introduction to the JSON:API specification in Drupal
Stop the noise! - Introduction to the JSON:API specification in DrupalStop the noise! - Introduction to the JSON:API specification in Drupal
Stop the noise! - Introduction to the JSON:API specification in Drupal
 
Fire-fighting java big data problems
Fire-fighting java big data problemsFire-fighting java big data problems
Fire-fighting java big data problems
 
Rapid Prototyping with Solr
Rapid Prototyping with SolrRapid Prototyping with Solr
Rapid Prototyping with Solr
 
Customer Intelligence: Using the ELK Stack to Analyze ForgeRock OpenAM Audit ...
Customer Intelligence: Using the ELK Stack to Analyze ForgeRock OpenAM Audit ...Customer Intelligence: Using the ELK Stack to Analyze ForgeRock OpenAM Audit ...
Customer Intelligence: Using the ELK Stack to Analyze ForgeRock OpenAM Audit ...
 
Ldap Synchronization Connector @ 2011.RMLL
Ldap Synchronization Connector @ 2011.RMLLLdap Synchronization Connector @ 2011.RMLL
Ldap Synchronization Connector @ 2011.RMLL
 
OData: Universal Data Solvent or Clunky Enterprise Goo? (GlueCon 2015)
OData: Universal Data Solvent or Clunky Enterprise Goo? (GlueCon 2015)OData: Universal Data Solvent or Clunky Enterprise Goo? (GlueCon 2015)
OData: Universal Data Solvent or Clunky Enterprise Goo? (GlueCon 2015)
 

Recently uploaded

Slack Application Development 101 Slides
Slack Application Development 101 SlidesSlack Application Development 101 Slides
Slack Application Development 101 Slidespraypatel2
 
Breaking the Kubernetes Kill Chain: Host Path Mount
Breaking the Kubernetes Kill Chain: Host Path MountBreaking the Kubernetes Kill Chain: Host Path Mount
Breaking the Kubernetes Kill Chain: Host Path MountPuma Security, LLC
 
🐬 The future of MySQL is Postgres 🐘
🐬  The future of MySQL is Postgres   🐘🐬  The future of MySQL is Postgres   🐘
🐬 The future of MySQL is Postgres 🐘RTylerCroy
 
EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptx
EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptxEIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptx
EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptxEarley Information Science
 
The 7 Things I Know About Cyber Security After 25 Years | April 2024
The 7 Things I Know About Cyber Security After 25 Years | April 2024The 7 Things I Know About Cyber Security After 25 Years | April 2024
The 7 Things I Know About Cyber Security After 25 Years | April 2024Rafal Los
 
The Codex of Business Writing Software for Real-World Solutions 2.pptx
The Codex of Business Writing Software for Real-World Solutions 2.pptxThe Codex of Business Writing Software for Real-World Solutions 2.pptx
The Codex of Business Writing Software for Real-World Solutions 2.pptxMalak Abu Hammad
 
Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024
Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024
Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024The Digital Insurer
 
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...Drew Madelung
 
Data Cloud, More than a CDP by Matt Robison
Data Cloud, More than a CDP by Matt RobisonData Cloud, More than a CDP by Matt Robison
Data Cloud, More than a CDP by Matt RobisonAnna Loughnan Colquhoun
 
Driving Behavioral Change for Information Management through Data-Driven Gree...
Driving Behavioral Change for Information Management through Data-Driven Gree...Driving Behavioral Change for Information Management through Data-Driven Gree...
Driving Behavioral Change for Information Management through Data-Driven Gree...Enterprise Knowledge
 
Finology Group – Insurtech Innovation Award 2024
Finology Group – Insurtech Innovation Award 2024Finology Group – Insurtech Innovation Award 2024
Finology Group – Insurtech Innovation Award 2024The Digital Insurer
 
How to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected WorkerHow to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected WorkerThousandEyes
 
Scaling API-first – The story of a global engineering organization
Scaling API-first – The story of a global engineering organizationScaling API-first – The story of a global engineering organization
Scaling API-first – The story of a global engineering organizationRadu Cotescu
 
Real Time Object Detection Using Open CV
Real Time Object Detection Using Open CVReal Time Object Detection Using Open CV
Real Time Object Detection Using Open CVKhem
 
04-2024-HHUG-Sales-and-Marketing-Alignment.pptx
04-2024-HHUG-Sales-and-Marketing-Alignment.pptx04-2024-HHUG-Sales-and-Marketing-Alignment.pptx
04-2024-HHUG-Sales-and-Marketing-Alignment.pptxHampshireHUG
 
What Are The Drone Anti-jamming Systems Technology?
What Are The Drone Anti-jamming Systems Technology?What Are The Drone Anti-jamming Systems Technology?
What Are The Drone Anti-jamming Systems Technology?Antenna Manufacturer Coco
 
How to convert PDF to text with Nanonets
How to convert PDF to text with NanonetsHow to convert PDF to text with Nanonets
How to convert PDF to text with Nanonetsnaman860154
 
Presentation on how to chat with PDF using ChatGPT code interpreter
Presentation on how to chat with PDF using ChatGPT code interpreterPresentation on how to chat with PDF using ChatGPT code interpreter
Presentation on how to chat with PDF using ChatGPT code interpreternaman860154
 
08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
08448380779 Call Girls In Diplomatic Enclave Women Seeking Men08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
08448380779 Call Girls In Diplomatic Enclave Women Seeking MenDelhi Call girls
 
CNv6 Instructor Chapter 6 Quality of Service
CNv6 Instructor Chapter 6 Quality of ServiceCNv6 Instructor Chapter 6 Quality of Service
CNv6 Instructor Chapter 6 Quality of Servicegiselly40
 

Recently uploaded (20)

Slack Application Development 101 Slides
Slack Application Development 101 SlidesSlack Application Development 101 Slides
Slack Application Development 101 Slides
 
Breaking the Kubernetes Kill Chain: Host Path Mount
Breaking the Kubernetes Kill Chain: Host Path MountBreaking the Kubernetes Kill Chain: Host Path Mount
Breaking the Kubernetes Kill Chain: Host Path Mount
 
🐬 The future of MySQL is Postgres 🐘
🐬  The future of MySQL is Postgres   🐘🐬  The future of MySQL is Postgres   🐘
🐬 The future of MySQL is Postgres 🐘
 
EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptx
EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptxEIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptx
EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptx
 
The 7 Things I Know About Cyber Security After 25 Years | April 2024
The 7 Things I Know About Cyber Security After 25 Years | April 2024The 7 Things I Know About Cyber Security After 25 Years | April 2024
The 7 Things I Know About Cyber Security After 25 Years | April 2024
 
The Codex of Business Writing Software for Real-World Solutions 2.pptx
The Codex of Business Writing Software for Real-World Solutions 2.pptxThe Codex of Business Writing Software for Real-World Solutions 2.pptx
The Codex of Business Writing Software for Real-World Solutions 2.pptx
 
Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024
Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024
Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024
 
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
 
Data Cloud, More than a CDP by Matt Robison
Data Cloud, More than a CDP by Matt RobisonData Cloud, More than a CDP by Matt Robison
Data Cloud, More than a CDP by Matt Robison
 
Driving Behavioral Change for Information Management through Data-Driven Gree...
Driving Behavioral Change for Information Management through Data-Driven Gree...Driving Behavioral Change for Information Management through Data-Driven Gree...
Driving Behavioral Change for Information Management through Data-Driven Gree...
 
Finology Group – Insurtech Innovation Award 2024
Finology Group – Insurtech Innovation Award 2024Finology Group – Insurtech Innovation Award 2024
Finology Group – Insurtech Innovation Award 2024
 
How to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected WorkerHow to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected Worker
 
Scaling API-first – The story of a global engineering organization
Scaling API-first – The story of a global engineering organizationScaling API-first – The story of a global engineering organization
Scaling API-first – The story of a global engineering organization
 
Real Time Object Detection Using Open CV
Real Time Object Detection Using Open CVReal Time Object Detection Using Open CV
Real Time Object Detection Using Open CV
 
04-2024-HHUG-Sales-and-Marketing-Alignment.pptx
04-2024-HHUG-Sales-and-Marketing-Alignment.pptx04-2024-HHUG-Sales-and-Marketing-Alignment.pptx
04-2024-HHUG-Sales-and-Marketing-Alignment.pptx
 
What Are The Drone Anti-jamming Systems Technology?
What Are The Drone Anti-jamming Systems Technology?What Are The Drone Anti-jamming Systems Technology?
What Are The Drone Anti-jamming Systems Technology?
 
How to convert PDF to text with Nanonets
How to convert PDF to text with NanonetsHow to convert PDF to text with Nanonets
How to convert PDF to text with Nanonets
 
Presentation on how to chat with PDF using ChatGPT code interpreter
Presentation on how to chat with PDF using ChatGPT code interpreterPresentation on how to chat with PDF using ChatGPT code interpreter
Presentation on how to chat with PDF using ChatGPT code interpreter
 
08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
08448380779 Call Girls In Diplomatic Enclave Women Seeking Men08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
 
CNv6 Instructor Chapter 6 Quality of Service
CNv6 Instructor Chapter 6 Quality of ServiceCNv6 Instructor Chapter 6 Quality of Service
CNv6 Instructor Chapter 6 Quality of Service
 

Not Big Data, AnyData

  • 2. Halo effects of a good incident response process: ¡ Fight FUD with truth ¡ IT ops love truth, therefore ops loves security, therefore security gets access to data/ resources ¡ When security=visibility and security!=NO, good things happen WHY ARE WE DOING IR?
  • 4. ¡ IR isn’t done until you’ve answered “why” ¡ Why=hard ¡ Need to fully understand threat motives via events and artifacts ¡ This requires quickly answering tough questions, which means searching, summarizing, and drilling down CONTEXT IS THE GOAL
  • 5. ¡ Search, summarize, drill-down on events and artifacts ¡ Leverages both local and global data ¡ Parallax of heterogeneous data sources describing the same action ELSA PROVIDES CONTEXT Parallax: derive depth from different points of view
  • 6. EVENT PARALLAX 1.1.1.1 <-> 2.2.2.2 Firewall • 200000 bytes downloaded Proxy • GET example.com/ evil.exe • category=uncategorized Bro • MD5 deadbeefdeadbeef • GET example.com/ evil.exe • Country code=RU IDS • Packed executable downloaded Windows • User is admin
  • 7. CONTEXT VIA TIMELINE 9/12 23:00 81.x.x.x GET company.com/directory.html 9/13 08:00 Email from admin@throwaway.com 9/13 09:05 Firewall connection 82.x.x.x 9/13 09:05 GET request to evil.com 82.x.x.x 9/13 09:05 Bro file MD5 from evil.com 82.x.x.x 9/13 09:05 IDS Packed Executable 82.x.x.x 9/14 07:00 IDS outbound RAR 83.x.x.x Recon Exploit Exfil Phish
  • 8. ELSA searches to construct the timeline: 1.  Find exploit: sig_msg:packed groupby:dstip | subsearch(! class:url groupby:site,dstip) | subsearch(! category:uncategorized! groupby:srcip,dstip)! BUILDING THE TIMELINE
  • 9. This gives us the attacker IP 82.x.x.x, victim IP 1.1.1.1. Continue with victim-centric searches. 2.  Find exfil: 1.1.1.1 groupby:sig_msg! 3.  Find bait: 1.1.1.1 groupby:email | subsearch(class:bro_smtp groupby:subject)! 4.  Find recon: site:company.com uri:directory.html! BUILDING THE TIMELINE
  • 10. 1.1.1.1 groupby:dstip | subsearch(class:url,srcip)! Results from 1.1.1.1 groupby:dstip: Subsearch evaluates: class:url srcip:2.2.2.2 OR srcip: 3.3.3.3! SUBSEARCHES Groupby Count 2.2.2.2 5 3.3.3.3 1
  • 12. Parse • Syslog-NG Normalize • elsa.pl Forward • ELSA plugins Archive • MySQL Index • Sphinx (indexer) INDEX DATA FLOW
  • 13. Parse • ELSA QueryParser.pm Query • Sphinx (searchd) Get Rows • MySQL Transform • ELSA plugins Connect • ELSA plugins SEARCH DATA FLOW
  • 15. ¡ Parsers are XML that go in /etc/elsa/ patterns.d/ ¡ ELSA install.sh will merge all files there into merged.xml, which is what syslog-ng uses ¡ Documentation online at ELSA’s Google Code project page WRITING PARSERS
  • 16. Syslog (RFC 3164) message format: <priority|facility> <timestamp> <program>: <message> <13> Sep 14 01:02:03 crond[1234]: Started! WRITING PARSERS
  • 17. Example: <ruleset id=“…”> <pattern>program</pattern> <rules> <rule id=“…” class=“integer class ID”> <patterns> <pattern>matching code</pattern> WRITING PARSERS
  • 18. Parser types: ¡ ESTRING: Slurp until given char or chars ¡ QSTRING: Slurp between chars ¡ NUMBER: Slurp digits ¡ The rest are not really needed WRITING PARSERS
  • 19. @<parser type>:<fieldname>:<chars>@! Parse: http://example.com/ with:! http://@ESTRING:site_name:/@! Extracts “example.com” to field “site_name” http:/@QSTRING:site_name:/@! Same but slower PARSER COMPONENT STRUCTURE
  • 20. Always try to use ESTRING http://example.com/! What if scheme is “https” or “ftp?” @ESTRING:://@@ESTRING:site_name:/@! First ESTRING moves pointer to after the first double slashes it finds, but does not extract a field name. PARSER STRATEGIES
  • 21. If you don’t have a character to start the parsing evaluation, you may get unexpected results, e.g.: my firewall log URL: http://example.com/! This matches as well, which may not be desired behavior. You can work around this with <pattern>program</pattern> in the <ruleset> element. ESTRING GOTCHAS
  • 22. Field names are stored in an abstracted format: Names map to field_order in SQL schema, class_id dependent INTEGRATING PARSERS WITH ELSA Integer name i0 i1 i2 i3 i4 i5 Field_order 5 6 7 8 9 10 String name s0 s1 s2 s3 s4 s5 Field_order 11 12 13 14 15 16
  • 24. <rule> … <examples> <example> <test_message program=“prog”>test text <test_values> <test_value name=“s0”>expected val ! pdbtool test merged.xml! TESTING PARSERS
  • 27. Goal: Provide ELSA access to an HR database to augment queries Desired ELSA query: datasource:hr department:finance groupby:username! Finds all users in finance! EXAMPLE: HR DATABASE PLUGIN
  • 28. Step 1: Define the SQL query on the HRIS: SELECT user, department FROM users! Step 2: Add to elsa_web.conf: “datasources”: {! “database”: {! “hr”: {! “dsn”: …! “username”: …! “password”: …! EXAMPLE: HR DATABASE PLUGIN
  • 29. ..continued: “fields”: [! { “name”: “user” },! { “name”: “department” }! ],! “query_template”: “SELECT %s FROM users WHERE %s %s”! }! EXAMPLE: HR DATABASE PLUGIN
  • 30. The query template will be built based on the ELSA query parameters, e.g. ELSA query: user:bob! ! Becomes SQL: WHERE user LIKE “%bob%”! EXAMPLE: HR DATABASE PLUGIN
  • 31. Let’s use it! datasource:hr department:finance groupby:user | ! subsearch(class:vpn groupby:srcip,user) |! whois | sum(description)! EXAMPLE: HR DATABASE PLUGIN Users in finance department Correlate with VPN logins by user Add srcip description Summarize by description
  • 32. Yields: EXAMPLE: HR DATABASE PLUGIN Description Count Comcast Corporation 234 Starbucks, Inc. 10 University of Lagos 3What!?! à
  • 33. Quantcast provides the top one million most visited sites. It would be nice to know when downloads occur from sites not on that list. Step 1: Grab the data: Top 1 million at https://www.quantcast.com/top-sites Rank Site 1 google.com 2 youtube.com 3 facebook.com 4 msn.com 5 twitter.com 6 wordpress.com 7 amazon.com 8 ebay.com 9 yahoo.com 10 yelp.com EXAMPLE: QUANTCAST TRANSFORM
  • 34. Step 2: Load the data into MySQL CREATE TABLE quantcast (count INT UNSIGNED, site VARCHAR(255));! LOAD DATA LOCAL INFILE “Quantcast.txt”! INTO TABLE quantcast IGNORE 6 LINES;! EXAMPLE: QUANTCAST TRANSFORM
  • 35. Step 3: Configure the transform {! “transforms”: {! “database”: {! “quantcast”: {! ! (dsn/user/pass/fields! query_template)! EXAMPLE: QUANTCAST TRANSFORM
  • 36. Step 4: Profit! Find downloads from uncommon sites (rank greater than ten thousand):! md5 class:bro_file groupby:site |! quantcast | sum(count) | has(10000)! EXAMPLE: QUANTCAST TRANSFORM
  • 37. There are many places you can add customization to ELSA. Some plugin ideas: ¡ Connector for ticketing system ¡ LDAP info ¡ Encrypted export ¡ Transform to check if user is logged in ¡ Transform to launch AR Drone attack THIS IS JUST A START!
  • 38. MORE INFORMATION Blog ossectools.blogspot.com Project Site enterprise-log-search-and- archive.googlecode.com Mailing List enterprise-log-search-and- archive.googlegroups.com Twitter @mcholste