SlideShare a Scribd company logo
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. HP Confidential.
Threat Analytics and
Visualization Solutions
for Big Data Security
Marco Casassa Mont, Yolanta Beres, Shane Sullivan & HPL BD4S
team
Security and Cloud Lab, HP Labs
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. HP Confidential.2
Outline
1.Motivation and Opportunity for HP
2.Use Case with DNS
3.The scale of the Problem
4.Our Solution
5.Demonstration
6.Conclusions
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. HP Confidential.3
Motivation
Identify Security Threats from Big Security
Data
Use Case: DNS Data
• Big Volume of Security Events …
• Gold mine for Security Information
• Hard to collect and analyze
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. HP Confidential.4
DNS is an important vector of attack
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. HP Confidential.5
What is DNS?
Local
DNS Server
Root
DNS Server
.com
DNS Server
company.com
DNS Server
Check for Zone
Check Cache
DNS Traffic Generated by:
- Users (e.g. browsing web)
- Applications, Servers, etc.
Endpoints
8. REPLY: 58.25.88.90 5. REPLY: ask “company.com”
4. QUERY: service.company.com?1. QUERY: service.company.com?
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. HP Confidential.6
1
10
100
1000
10000
100000
1000000
Routers VPN McAfee ePO Active Directory Web Proxy DNS
Eventspersecond(logarithmicscale)
The Scale of DNS Data
0
20000
40000
60000
80000
100000
120000
140000
Routers VPN McAfee ePO Active Directory Web Proxy DNS
Eventspersecond(linearscale)
HP IT Pilot: 120,000 DNS events/second
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. HP Confidential.7
What is the Problem?
Scale
• DNS is one of
the most
voluminous
sources of
events in the
enterprise
Information
Granularity
• Traditional
DNS servers
do not log
enough detail
to detect
many kinds
of attacks
Analysis
• Limited
analysis tools
on top of
event
collection for
DNS
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. HP Confidential.9
End to end handling of DNS events
Our Solution
Collection Storage Analysis Visualization
Filtering out
90-95%
DNS traffic
- Real-Time
- Near-Time/
Historical
- Dashboards
- Graphs
- Timelines …
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. HP Confidential.10
DNS Event Processing: Architecture
DNS Server(s)
DNS
Packet
Capture
Whitelist
Network tapDNS queries
and responses
ArcSight
Logger
ArcSight
ESM
Blacklist
HPL Analytical and
Visualization
Solution
Event
Logging
Correlation &
Alerting
Blacklist
&
Whitelist
Manager
Real-Time Processing
Near-time/
Historical Analysis
DNS events:
Queries & Replies
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. HP Confidential.11
DNS
Packet
Capturer
Security Event Logs
Scheduler & Workflows
Vertica
Anomaly
Detectio
n
Threat
Indicators
Threat Analytics &
Visualization
Library
HPL Security Analytical &
Visualization
Network Systems
Visual
Processin
g
ArcSight
ESM
Filtered
DNS events
Event
Downloader
&
Processor
ArcSight
Logger
Events
ESM Alerts
Whitelist,
RepSM Blacklist
Generator
Real-time Analysis Near-time/Historical Analysis
Hadoop,
Autonomy, R
Web Server
Threat
Intelligenc
e
Anomalies,
Threat Findings
Anomalies, Threats, Graphs
Syslog
Server
Alerts Introspection
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. HP Confidential.12
Analytic Approaches
Detection of Bad Devices and Domain Names:
– Large #Queries to Black-listed Domain Names
– Anomalous Traffic Patterns
– Usage of Machine-generated Domain Names
– Timeseries Analysis
Success Cases:
CryptoLockers, Zeus-Variant, C&C, etc.
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. HP Confidential.13
Analytic Approaches
• Mechanisms
• ArcSight ESM rules
• Vertica SQL Queries
• Algorithms & Code
• Pipeline/Distribution of Queries and Code
• Pilots with HP Labs IT and HP Cybersecurity
Defence Center (CDC)
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. HP Confidential.14
Data Visualization Approaches
Connected graphs
Time series
Raw data exploration
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. HP Confidential.
Demo …
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. HP Confidential.16
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. HP Confidential.17
Current Status and Next Steps
• 2 Pilots: HP Labs IT, HP IT
• Lot of HP customers’ interest: 1 pilot to be
finalized
• Collaborations:
HP ESP, TippingPoint/DVLabs, HP ESS, HP IT,
HPL IT
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. HP Confidential.
Thank you
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. HP Confidential.19
HP HAVEn – Big Data platform
HAVEn
Social media IT/OT ImagesAudioVideo
Transactional
dataMobile Search engineEmail Texts
Catalog massive
volumes of
distributed data
Hadoop/
HDFS
Process and
index all
information
Autonomy
IDOL
Analyze at
extreme scale
in real-time
Vertica
Collect & unify
machine data
Enterprise
Security
Powering
HP Software
+ your apps
nApps
Documents
hp.com/haven
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. HP Confidential.20
Competitive Landscape
Main HP Competitors in the
Security Market already making
claims about the following features:
• Network Packet Capture
capabilities, inclusive of DNS
traffic
• Back-end advanced Analytics on
large amounts of (security)
events, complementing real-time
event/alert processing

More Related Content

What's hot

Red Hat in Financial Services - Presentation at Hortonworks Booth - Strata 2014
Red Hat in Financial Services - Presentation at Hortonworks Booth - Strata 2014Red Hat in Financial Services - Presentation at Hortonworks Booth - Strata 2014
Red Hat in Financial Services - Presentation at Hortonworks Booth - Strata 2014
Hortonworks
 
Hortonworks - What's Possible with a Modern Data Architecture?
Hortonworks - What's Possible with a Modern Data Architecture?Hortonworks - What's Possible with a Modern Data Architecture?
Hortonworks - What's Possible with a Modern Data Architecture?
Hortonworks
 
Capture Discovery
Capture DiscoveryCapture Discovery
Capture Discovery
wlucina
 
4 Essential Steps for Managing Sensitive Data
4 Essential Steps for Managing Sensitive Data4 Essential Steps for Managing Sensitive Data
4 Essential Steps for Managing Sensitive Data
Hortonworks
 
How to use flash drives with Apache Hadoop 3.x: Real world use cases and proo...
How to use flash drives with Apache Hadoop 3.x: Real world use cases and proo...How to use flash drives with Apache Hadoop 3.x: Real world use cases and proo...
How to use flash drives with Apache Hadoop 3.x: Real world use cases and proo...
DataWorks Summit
 
AWS Public Sector Symposium 2014 Canberra | Secure Hadoop as a Service
AWS Public Sector Symposium 2014 Canberra | Secure Hadoop as a ServiceAWS Public Sector Symposium 2014 Canberra | Secure Hadoop as a Service
AWS Public Sector Symposium 2014 Canberra | Secure Hadoop as a Service
Amazon Web Services
 
Hadoop: The Unintended Benefits
Hadoop: The Unintended BenefitsHadoop: The Unintended Benefits
Hadoop: The Unintended Benefits
DataWorks Summit
 
Discover HDP 2.1: Apache Falcon for Data Governance in Hadoop
Discover HDP 2.1: Apache Falcon for Data Governance in HadoopDiscover HDP 2.1: Apache Falcon for Data Governance in Hadoop
Discover HDP 2.1: Apache Falcon for Data Governance in Hadoop
Hortonworks
 
Sqrrl Overview for Stac Research
Sqrrl Overview for Stac ResearchSqrrl Overview for Stac Research
Sqrrl Overview for Stac Research
Sqrrl
 
Apache Hadoop Crash Course - HS16SJ
Apache Hadoop Crash Course - HS16SJApache Hadoop Crash Course - HS16SJ
Apache Hadoop Crash Course - HS16SJ
DataWorks Summit/Hadoop Summit
 
Sqrrl February Webinar: Breaking Down Data Silos
Sqrrl February Webinar: Breaking Down Data SilosSqrrl February Webinar: Breaking Down Data Silos
Sqrrl February Webinar: Breaking Down Data Silos
Sqrrl
 
Data Science with Apache Spark - Crash Course - HS16SJ
Data Science with Apache Spark - Crash Course - HS16SJData Science with Apache Spark - Crash Course - HS16SJ
Data Science with Apache Spark - Crash Course - HS16SJ
DataWorks Summit/Hadoop Summit
 
Atul Mithe
Atul MitheAtul Mithe
Atul Mithe
Atul Mithe
 
2014 july 24_what_ishadoop
2014 july 24_what_ishadoop2014 july 24_what_ishadoop
2014 july 24_what_ishadoop
Adam Muise
 
REAL-TIME INGESTING AND TRANSFORMING SENSOR DATA & SOCIAL DATA w/ NIFI + TENS...
REAL-TIME INGESTING AND TRANSFORMING SENSOR DATA & SOCIAL DATA w/ NIFI + TENS...REAL-TIME INGESTING AND TRANSFORMING SENSOR DATA & SOCIAL DATA w/ NIFI + TENS...
REAL-TIME INGESTING AND TRANSFORMING SENSOR DATA & SOCIAL DATA w/ NIFI + TENS...
Timothy Spann
 
Security Updates: More Seamless Access Controls with Apache Spark and Apache ...
Security Updates: More Seamless Access Controls with Apache Spark and Apache ...Security Updates: More Seamless Access Controls with Apache Spark and Apache ...
Security Updates: More Seamless Access Controls with Apache Spark and Apache ...
DataWorks Summit
 
Interactive real-time dashboards on data streams using Kafka, Druid, and Supe...
Interactive real-time dashboards on data streams using Kafka, Druid, and Supe...Interactive real-time dashboards on data streams using Kafka, Druid, and Supe...
Interactive real-time dashboards on data streams using Kafka, Druid, and Supe...
DataWorks Summit
 
Hortonworks Yarn Code Walk Through January 2014
Hortonworks Yarn Code Walk Through January 2014Hortonworks Yarn Code Walk Through January 2014
Hortonworks Yarn Code Walk Through January 2014
Hortonworks
 
Architecting a multi-tenanted platform
Architecting a multi-tenanted platform Architecting a multi-tenanted platform
Architecting a multi-tenanted platform
DataWorks Summit/Hadoop Summit
 
CCD-410 Cloudera Study Material
CCD-410 Cloudera Study MaterialCCD-410 Cloudera Study Material
CCD-410 Cloudera Study Material
Roxycodone Online
 

What's hot (20)

Red Hat in Financial Services - Presentation at Hortonworks Booth - Strata 2014
Red Hat in Financial Services - Presentation at Hortonworks Booth - Strata 2014Red Hat in Financial Services - Presentation at Hortonworks Booth - Strata 2014
Red Hat in Financial Services - Presentation at Hortonworks Booth - Strata 2014
 
Hortonworks - What's Possible with a Modern Data Architecture?
Hortonworks - What's Possible with a Modern Data Architecture?Hortonworks - What's Possible with a Modern Data Architecture?
Hortonworks - What's Possible with a Modern Data Architecture?
 
Capture Discovery
Capture DiscoveryCapture Discovery
Capture Discovery
 
4 Essential Steps for Managing Sensitive Data
4 Essential Steps for Managing Sensitive Data4 Essential Steps for Managing Sensitive Data
4 Essential Steps for Managing Sensitive Data
 
How to use flash drives with Apache Hadoop 3.x: Real world use cases and proo...
How to use flash drives with Apache Hadoop 3.x: Real world use cases and proo...How to use flash drives with Apache Hadoop 3.x: Real world use cases and proo...
How to use flash drives with Apache Hadoop 3.x: Real world use cases and proo...
 
AWS Public Sector Symposium 2014 Canberra | Secure Hadoop as a Service
AWS Public Sector Symposium 2014 Canberra | Secure Hadoop as a ServiceAWS Public Sector Symposium 2014 Canberra | Secure Hadoop as a Service
AWS Public Sector Symposium 2014 Canberra | Secure Hadoop as a Service
 
Hadoop: The Unintended Benefits
Hadoop: The Unintended BenefitsHadoop: The Unintended Benefits
Hadoop: The Unintended Benefits
 
Discover HDP 2.1: Apache Falcon for Data Governance in Hadoop
Discover HDP 2.1: Apache Falcon for Data Governance in HadoopDiscover HDP 2.1: Apache Falcon for Data Governance in Hadoop
Discover HDP 2.1: Apache Falcon for Data Governance in Hadoop
 
Sqrrl Overview for Stac Research
Sqrrl Overview for Stac ResearchSqrrl Overview for Stac Research
Sqrrl Overview for Stac Research
 
Apache Hadoop Crash Course - HS16SJ
Apache Hadoop Crash Course - HS16SJApache Hadoop Crash Course - HS16SJ
Apache Hadoop Crash Course - HS16SJ
 
Sqrrl February Webinar: Breaking Down Data Silos
Sqrrl February Webinar: Breaking Down Data SilosSqrrl February Webinar: Breaking Down Data Silos
Sqrrl February Webinar: Breaking Down Data Silos
 
Data Science with Apache Spark - Crash Course - HS16SJ
Data Science with Apache Spark - Crash Course - HS16SJData Science with Apache Spark - Crash Course - HS16SJ
Data Science with Apache Spark - Crash Course - HS16SJ
 
Atul Mithe
Atul MitheAtul Mithe
Atul Mithe
 
2014 july 24_what_ishadoop
2014 july 24_what_ishadoop2014 july 24_what_ishadoop
2014 july 24_what_ishadoop
 
REAL-TIME INGESTING AND TRANSFORMING SENSOR DATA & SOCIAL DATA w/ NIFI + TENS...
REAL-TIME INGESTING AND TRANSFORMING SENSOR DATA & SOCIAL DATA w/ NIFI + TENS...REAL-TIME INGESTING AND TRANSFORMING SENSOR DATA & SOCIAL DATA w/ NIFI + TENS...
REAL-TIME INGESTING AND TRANSFORMING SENSOR DATA & SOCIAL DATA w/ NIFI + TENS...
 
Security Updates: More Seamless Access Controls with Apache Spark and Apache ...
Security Updates: More Seamless Access Controls with Apache Spark and Apache ...Security Updates: More Seamless Access Controls with Apache Spark and Apache ...
Security Updates: More Seamless Access Controls with Apache Spark and Apache ...
 
Interactive real-time dashboards on data streams using Kafka, Druid, and Supe...
Interactive real-time dashboards on data streams using Kafka, Druid, and Supe...Interactive real-time dashboards on data streams using Kafka, Druid, and Supe...
Interactive real-time dashboards on data streams using Kafka, Druid, and Supe...
 
Hortonworks Yarn Code Walk Through January 2014
Hortonworks Yarn Code Walk Through January 2014Hortonworks Yarn Code Walk Through January 2014
Hortonworks Yarn Code Walk Through January 2014
 
Architecting a multi-tenanted platform
Architecting a multi-tenanted platform Architecting a multi-tenanted platform
Architecting a multi-tenanted platform
 
CCD-410 Cloudera Study Material
CCD-410 Cloudera Study MaterialCCD-410 Cloudera Study Material
CCD-410 Cloudera Study Material
 

Similar to Big Data for Security - Threat Analytics

Security intelligence using big data presentation (engineering seminar)
Security intelligence using big data  presentation (engineering seminar)Security intelligence using big data  presentation (engineering seminar)
Security intelligence using big data presentation (engineering seminar)
Marco Casassa Mont
 
Big Data for Security
Big Data for SecurityBig Data for Security
Big Data for Security
Marco Casassa Mont
 
HP Vertica and MapR Webinar: Building a Business Case for SQL-on-Hadoop
HP Vertica and MapR Webinar: Building a Business Case for SQL-on-HadoopHP Vertica and MapR Webinar: Building a Business Case for SQL-on-Hadoop
HP Vertica and MapR Webinar: Building a Business Case for SQL-on-Hadoop
MapR Technologies
 
Hortonworks sqrrl webinar v5.pptx
Hortonworks sqrrl webinar v5.pptxHortonworks sqrrl webinar v5.pptx
Hortonworks sqrrl webinar v5.pptx
Hortonworks
 
Hortonworks and Voltage Security webinar
Hortonworks and Voltage Security webinarHortonworks and Voltage Security webinar
Hortonworks and Voltage Security webinar
Hortonworks
 
To Serve and Protect: Making Sense of Hadoop Security
To Serve and Protect: Making Sense of Hadoop Security To Serve and Protect: Making Sense of Hadoop Security
To Serve and Protect: Making Sense of Hadoop Security
Inside Analysis
 
Supporting Financial Services with a More Flexible Approach to Big Data
Supporting Financial Services with a More Flexible Approach to Big DataSupporting Financial Services with a More Flexible Approach to Big Data
Supporting Financial Services with a More Flexible Approach to Big Data
WANdisco Plc
 
Create a Smarter Data Lake with HP Haven and Apache Hadoop
Create a Smarter Data Lake with HP Haven and Apache HadoopCreate a Smarter Data Lake with HP Haven and Apache Hadoop
Create a Smarter Data Lake with HP Haven and Apache Hadoop
Hortonworks
 
HP Enterprise Software: Making your applications and information work for you
HP Enterprise Software: Making your applications and information work for youHP Enterprise Software: Making your applications and information work for you
HP Enterprise Software: Making your applications and information work for you
HP Enterprise Italia
 
Enterprise Hadoop with Hortonworks and Nimble Storage
Enterprise Hadoop with Hortonworks and Nimble StorageEnterprise Hadoop with Hortonworks and Nimble Storage
Enterprise Hadoop with Hortonworks and Nimble Storage
Hortonworks
 
Hortonworks and Platfora in Financial Services - Webinar
Hortonworks and Platfora in Financial Services - WebinarHortonworks and Platfora in Financial Services - Webinar
Hortonworks and Platfora in Financial Services - Webinar
Hortonworks
 
Level Up – How to Achieve Hadoop Acceleration
Level Up – How to Achieve Hadoop AccelerationLevel Up – How to Achieve Hadoop Acceleration
Level Up – How to Achieve Hadoop Acceleration
Inside Analysis
 
Trafodion – an enterprise class sql based on hadoop
Trafodion – an enterprise class sql based on hadoopTrafodion – an enterprise class sql based on hadoop
Trafodion – an enterprise class sql based on hadoop
Krishna-Kumar
 
Big Data & SQL: The On-Ramp to Hadoop
Big Data & SQL: The On-Ramp to Hadoop Big Data & SQL: The On-Ramp to Hadoop
Big Data & SQL: The On-Ramp to Hadoop
Inside Analysis
 
Supporting Financial Services with a More Flexible Approach to Big Data
Supporting Financial Services with a More Flexible Approach to Big DataSupporting Financial Services with a More Flexible Approach to Big Data
Supporting Financial Services with a More Flexible Approach to Big Data
Hortonworks
 
A modern, flexible approach to Hadoop implementation incorporating innovation...
A modern, flexible approach to Hadoop implementation incorporating innovation...A modern, flexible approach to Hadoop implementation incorporating innovation...
A modern, flexible approach to Hadoop implementation incorporating innovation...
DataWorks Summit
 
Service Anywhere What's New March 2014
Service Anywhere What's New March 2014Service Anywhere What's New March 2014
Service Anywhere What's New March 2014
Pronq by HP
 
Up Your Analytics Game with Pentaho and Vertica
Up Your Analytics Game with Pentaho and Vertica Up Your Analytics Game with Pentaho and Vertica
Up Your Analytics Game with Pentaho and Vertica
Pentaho
 
Moustafa Soliman "HP Vertica- Solving Facebook Big Data challenges"
Moustafa Soliman "HP Vertica- Solving Facebook Big Data challenges" Moustafa Soliman "HP Vertica- Solving Facebook Big Data challenges"
Moustafa Soliman "HP Vertica- Solving Facebook Big Data challenges"
Dataconomy Media
 
A Comprehensive Approach to Building your Big Data - with Cisco, Hortonworks ...
A Comprehensive Approach to Building your Big Data - with Cisco, Hortonworks ...A Comprehensive Approach to Building your Big Data - with Cisco, Hortonworks ...
A Comprehensive Approach to Building your Big Data - with Cisco, Hortonworks ...
Hortonworks
 

Similar to Big Data for Security - Threat Analytics (20)

Security intelligence using big data presentation (engineering seminar)
Security intelligence using big data  presentation (engineering seminar)Security intelligence using big data  presentation (engineering seminar)
Security intelligence using big data presentation (engineering seminar)
 
Big Data for Security
Big Data for SecurityBig Data for Security
Big Data for Security
 
HP Vertica and MapR Webinar: Building a Business Case for SQL-on-Hadoop
HP Vertica and MapR Webinar: Building a Business Case for SQL-on-HadoopHP Vertica and MapR Webinar: Building a Business Case for SQL-on-Hadoop
HP Vertica and MapR Webinar: Building a Business Case for SQL-on-Hadoop
 
Hortonworks sqrrl webinar v5.pptx
Hortonworks sqrrl webinar v5.pptxHortonworks sqrrl webinar v5.pptx
Hortonworks sqrrl webinar v5.pptx
 
Hortonworks and Voltage Security webinar
Hortonworks and Voltage Security webinarHortonworks and Voltage Security webinar
Hortonworks and Voltage Security webinar
 
To Serve and Protect: Making Sense of Hadoop Security
To Serve and Protect: Making Sense of Hadoop Security To Serve and Protect: Making Sense of Hadoop Security
To Serve and Protect: Making Sense of Hadoop Security
 
Supporting Financial Services with a More Flexible Approach to Big Data
Supporting Financial Services with a More Flexible Approach to Big DataSupporting Financial Services with a More Flexible Approach to Big Data
Supporting Financial Services with a More Flexible Approach to Big Data
 
Create a Smarter Data Lake with HP Haven and Apache Hadoop
Create a Smarter Data Lake with HP Haven and Apache HadoopCreate a Smarter Data Lake with HP Haven and Apache Hadoop
Create a Smarter Data Lake with HP Haven and Apache Hadoop
 
HP Enterprise Software: Making your applications and information work for you
HP Enterprise Software: Making your applications and information work for youHP Enterprise Software: Making your applications and information work for you
HP Enterprise Software: Making your applications and information work for you
 
Enterprise Hadoop with Hortonworks and Nimble Storage
Enterprise Hadoop with Hortonworks and Nimble StorageEnterprise Hadoop with Hortonworks and Nimble Storage
Enterprise Hadoop with Hortonworks and Nimble Storage
 
Hortonworks and Platfora in Financial Services - Webinar
Hortonworks and Platfora in Financial Services - WebinarHortonworks and Platfora in Financial Services - Webinar
Hortonworks and Platfora in Financial Services - Webinar
 
Level Up – How to Achieve Hadoop Acceleration
Level Up – How to Achieve Hadoop AccelerationLevel Up – How to Achieve Hadoop Acceleration
Level Up – How to Achieve Hadoop Acceleration
 
Trafodion – an enterprise class sql based on hadoop
Trafodion – an enterprise class sql based on hadoopTrafodion – an enterprise class sql based on hadoop
Trafodion – an enterprise class sql based on hadoop
 
Big Data & SQL: The On-Ramp to Hadoop
Big Data & SQL: The On-Ramp to Hadoop Big Data & SQL: The On-Ramp to Hadoop
Big Data & SQL: The On-Ramp to Hadoop
 
Supporting Financial Services with a More Flexible Approach to Big Data
Supporting Financial Services with a More Flexible Approach to Big DataSupporting Financial Services with a More Flexible Approach to Big Data
Supporting Financial Services with a More Flexible Approach to Big Data
 
A modern, flexible approach to Hadoop implementation incorporating innovation...
A modern, flexible approach to Hadoop implementation incorporating innovation...A modern, flexible approach to Hadoop implementation incorporating innovation...
A modern, flexible approach to Hadoop implementation incorporating innovation...
 
Service Anywhere What's New March 2014
Service Anywhere What's New March 2014Service Anywhere What's New March 2014
Service Anywhere What's New March 2014
 
Up Your Analytics Game with Pentaho and Vertica
Up Your Analytics Game with Pentaho and Vertica Up Your Analytics Game with Pentaho and Vertica
Up Your Analytics Game with Pentaho and Vertica
 
Moustafa Soliman "HP Vertica- Solving Facebook Big Data challenges"
Moustafa Soliman "HP Vertica- Solving Facebook Big Data challenges" Moustafa Soliman "HP Vertica- Solving Facebook Big Data challenges"
Moustafa Soliman "HP Vertica- Solving Facebook Big Data challenges"
 
A Comprehensive Approach to Building your Big Data - with Cisco, Hortonworks ...
A Comprehensive Approach to Building your Big Data - with Cisco, Hortonworks ...A Comprehensive Approach to Building your Big Data - with Cisco, Hortonworks ...
A Comprehensive Approach to Building your Big Data - with Cisco, Hortonworks ...
 

Recently uploaded

STATATHON: Unleashing the Power of Statistics in a 48-Hour Knowledge Extravag...
STATATHON: Unleashing the Power of Statistics in a 48-Hour Knowledge Extravag...STATATHON: Unleashing the Power of Statistics in a 48-Hour Knowledge Extravag...
STATATHON: Unleashing the Power of Statistics in a 48-Hour Knowledge Extravag...
sameer shah
 
Open Source Contributions to Postgres: The Basics POSETTE 2024
Open Source Contributions to Postgres: The Basics POSETTE 2024Open Source Contributions to Postgres: The Basics POSETTE 2024
Open Source Contributions to Postgres: The Basics POSETTE 2024
ElizabethGarrettChri
 
Palo Alto Cortex XDR presentation .......
Palo Alto Cortex XDR presentation .......Palo Alto Cortex XDR presentation .......
Palo Alto Cortex XDR presentation .......
Sachin Paul
 
The Building Blocks of QuestDB, a Time Series Database
The Building Blocks of QuestDB, a Time Series DatabaseThe Building Blocks of QuestDB, a Time Series Database
The Building Blocks of QuestDB, a Time Series Database
javier ramirez
 
DSSML24_tspann_CodelessGenerativeAIPipelines
DSSML24_tspann_CodelessGenerativeAIPipelinesDSSML24_tspann_CodelessGenerativeAIPipelines
DSSML24_tspann_CodelessGenerativeAIPipelines
Timothy Spann
 
Global Situational Awareness of A.I. and where its headed
Global Situational Awareness of A.I. and where its headedGlobal Situational Awareness of A.I. and where its headed
Global Situational Awareness of A.I. and where its headed
vikram sood
 
原版制作(unimelb毕业证书)墨尔本大学毕业证Offer一模一样
原版制作(unimelb毕业证书)墨尔本大学毕业证Offer一模一样原版制作(unimelb毕业证书)墨尔本大学毕业证Offer一模一样
原版制作(unimelb毕业证书)墨尔本大学毕业证Offer一模一样
ihavuls
 
原版一比一利兹贝克特大学毕业证(LeedsBeckett毕业证书)如何办理
原版一比一利兹贝克特大学毕业证(LeedsBeckett毕业证书)如何办理原版一比一利兹贝克特大学毕业证(LeedsBeckett毕业证书)如何办理
原版一比一利兹贝克特大学毕业证(LeedsBeckett毕业证书)如何办理
wyddcwye1
 
一比一原版(UCSF文凭证书)旧金山分校毕业证如何办理
一比一原版(UCSF文凭证书)旧金山分校毕业证如何办理一比一原版(UCSF文凭证书)旧金山分校毕业证如何办理
一比一原版(UCSF文凭证书)旧金山分校毕业证如何办理
nuttdpt
 
Predictably Improve Your B2B Tech Company's Performance by Leveraging Data
Predictably Improve Your B2B Tech Company's Performance by Leveraging DataPredictably Improve Your B2B Tech Company's Performance by Leveraging Data
Predictably Improve Your B2B Tech Company's Performance by Leveraging Data
Kiwi Creative
 
Everything you wanted to know about LIHTC
Everything you wanted to know about LIHTCEverything you wanted to know about LIHTC
Everything you wanted to know about LIHTC
Roger Valdez
 
一比一原版(GWU,GW文凭证书)乔治·华盛顿大学毕业证如何办理
一比一原版(GWU,GW文凭证书)乔治·华盛顿大学毕业证如何办理一比一原版(GWU,GW文凭证书)乔治·华盛顿大学毕业证如何办理
一比一原版(GWU,GW文凭证书)乔治·华盛顿大学毕业证如何办理
bopyb
 
University of New South Wales degree offer diploma Transcript
University of New South Wales degree offer diploma TranscriptUniversity of New South Wales degree offer diploma Transcript
University of New South Wales degree offer diploma Transcript
soxrziqu
 
The Ipsos - AI - Monitor 2024 Report.pdf
The  Ipsos - AI - Monitor 2024 Report.pdfThe  Ipsos - AI - Monitor 2024 Report.pdf
The Ipsos - AI - Monitor 2024 Report.pdf
Social Samosa
 
一比一原版(UCSB文凭证书)圣芭芭拉分校毕业证如何办理
一比一原版(UCSB文凭证书)圣芭芭拉分校毕业证如何办理一比一原版(UCSB文凭证书)圣芭芭拉分校毕业证如何办理
一比一原版(UCSB文凭证书)圣芭芭拉分校毕业证如何办理
nuttdpt
 
4th Modern Marketing Reckoner by MMA Global India & Group M: 60+ experts on W...
4th Modern Marketing Reckoner by MMA Global India & Group M: 60+ experts on W...4th Modern Marketing Reckoner by MMA Global India & Group M: 60+ experts on W...
4th Modern Marketing Reckoner by MMA Global India & Group M: 60+ experts on W...
Social Samosa
 
Population Growth in Bataan: The effects of population growth around rural pl...
Population Growth in Bataan: The effects of population growth around rural pl...Population Growth in Bataan: The effects of population growth around rural pl...
Population Growth in Bataan: The effects of population growth around rural pl...
Bill641377
 
A presentation that explain the Power BI Licensing
A presentation that explain the Power BI LicensingA presentation that explain the Power BI Licensing
A presentation that explain the Power BI Licensing
AlessioFois2
 
Analysis insight about a Flyball dog competition team's performance
Analysis insight about a Flyball dog competition team's performanceAnalysis insight about a Flyball dog competition team's performance
Analysis insight about a Flyball dog competition team's performance
roli9797
 
Intelligence supported media monitoring in veterinary medicine
Intelligence supported media monitoring in veterinary medicineIntelligence supported media monitoring in veterinary medicine
Intelligence supported media monitoring in veterinary medicine
AndrzejJarynowski
 

Recently uploaded (20)

STATATHON: Unleashing the Power of Statistics in a 48-Hour Knowledge Extravag...
STATATHON: Unleashing the Power of Statistics in a 48-Hour Knowledge Extravag...STATATHON: Unleashing the Power of Statistics in a 48-Hour Knowledge Extravag...
STATATHON: Unleashing the Power of Statistics in a 48-Hour Knowledge Extravag...
 
Open Source Contributions to Postgres: The Basics POSETTE 2024
Open Source Contributions to Postgres: The Basics POSETTE 2024Open Source Contributions to Postgres: The Basics POSETTE 2024
Open Source Contributions to Postgres: The Basics POSETTE 2024
 
Palo Alto Cortex XDR presentation .......
Palo Alto Cortex XDR presentation .......Palo Alto Cortex XDR presentation .......
Palo Alto Cortex XDR presentation .......
 
The Building Blocks of QuestDB, a Time Series Database
The Building Blocks of QuestDB, a Time Series DatabaseThe Building Blocks of QuestDB, a Time Series Database
The Building Blocks of QuestDB, a Time Series Database
 
DSSML24_tspann_CodelessGenerativeAIPipelines
DSSML24_tspann_CodelessGenerativeAIPipelinesDSSML24_tspann_CodelessGenerativeAIPipelines
DSSML24_tspann_CodelessGenerativeAIPipelines
 
Global Situational Awareness of A.I. and where its headed
Global Situational Awareness of A.I. and where its headedGlobal Situational Awareness of A.I. and where its headed
Global Situational Awareness of A.I. and where its headed
 
原版制作(unimelb毕业证书)墨尔本大学毕业证Offer一模一样
原版制作(unimelb毕业证书)墨尔本大学毕业证Offer一模一样原版制作(unimelb毕业证书)墨尔本大学毕业证Offer一模一样
原版制作(unimelb毕业证书)墨尔本大学毕业证Offer一模一样
 
原版一比一利兹贝克特大学毕业证(LeedsBeckett毕业证书)如何办理
原版一比一利兹贝克特大学毕业证(LeedsBeckett毕业证书)如何办理原版一比一利兹贝克特大学毕业证(LeedsBeckett毕业证书)如何办理
原版一比一利兹贝克特大学毕业证(LeedsBeckett毕业证书)如何办理
 
一比一原版(UCSF文凭证书)旧金山分校毕业证如何办理
一比一原版(UCSF文凭证书)旧金山分校毕业证如何办理一比一原版(UCSF文凭证书)旧金山分校毕业证如何办理
一比一原版(UCSF文凭证书)旧金山分校毕业证如何办理
 
Predictably Improve Your B2B Tech Company's Performance by Leveraging Data
Predictably Improve Your B2B Tech Company's Performance by Leveraging DataPredictably Improve Your B2B Tech Company's Performance by Leveraging Data
Predictably Improve Your B2B Tech Company's Performance by Leveraging Data
 
Everything you wanted to know about LIHTC
Everything you wanted to know about LIHTCEverything you wanted to know about LIHTC
Everything you wanted to know about LIHTC
 
一比一原版(GWU,GW文凭证书)乔治·华盛顿大学毕业证如何办理
一比一原版(GWU,GW文凭证书)乔治·华盛顿大学毕业证如何办理一比一原版(GWU,GW文凭证书)乔治·华盛顿大学毕业证如何办理
一比一原版(GWU,GW文凭证书)乔治·华盛顿大学毕业证如何办理
 
University of New South Wales degree offer diploma Transcript
University of New South Wales degree offer diploma TranscriptUniversity of New South Wales degree offer diploma Transcript
University of New South Wales degree offer diploma Transcript
 
The Ipsos - AI - Monitor 2024 Report.pdf
The  Ipsos - AI - Monitor 2024 Report.pdfThe  Ipsos - AI - Monitor 2024 Report.pdf
The Ipsos - AI - Monitor 2024 Report.pdf
 
一比一原版(UCSB文凭证书)圣芭芭拉分校毕业证如何办理
一比一原版(UCSB文凭证书)圣芭芭拉分校毕业证如何办理一比一原版(UCSB文凭证书)圣芭芭拉分校毕业证如何办理
一比一原版(UCSB文凭证书)圣芭芭拉分校毕业证如何办理
 
4th Modern Marketing Reckoner by MMA Global India & Group M: 60+ experts on W...
4th Modern Marketing Reckoner by MMA Global India & Group M: 60+ experts on W...4th Modern Marketing Reckoner by MMA Global India & Group M: 60+ experts on W...
4th Modern Marketing Reckoner by MMA Global India & Group M: 60+ experts on W...
 
Population Growth in Bataan: The effects of population growth around rural pl...
Population Growth in Bataan: The effects of population growth around rural pl...Population Growth in Bataan: The effects of population growth around rural pl...
Population Growth in Bataan: The effects of population growth around rural pl...
 
A presentation that explain the Power BI Licensing
A presentation that explain the Power BI LicensingA presentation that explain the Power BI Licensing
A presentation that explain the Power BI Licensing
 
Analysis insight about a Flyball dog competition team's performance
Analysis insight about a Flyball dog competition team's performanceAnalysis insight about a Flyball dog competition team's performance
Analysis insight about a Flyball dog competition team's performance
 
Intelligence supported media monitoring in veterinary medicine
Intelligence supported media monitoring in veterinary medicineIntelligence supported media monitoring in veterinary medicine
Intelligence supported media monitoring in veterinary medicine
 

Big Data for Security - Threat Analytics

  • 1. © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. HP Confidential. Threat Analytics and Visualization Solutions for Big Data Security Marco Casassa Mont, Yolanta Beres, Shane Sullivan & HPL BD4S team Security and Cloud Lab, HP Labs
  • 2. © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. HP Confidential.2 Outline 1.Motivation and Opportunity for HP 2.Use Case with DNS 3.The scale of the Problem 4.Our Solution 5.Demonstration 6.Conclusions
  • 3. © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. HP Confidential.3 Motivation Identify Security Threats from Big Security Data Use Case: DNS Data • Big Volume of Security Events … • Gold mine for Security Information • Hard to collect and analyze
  • 4. © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. HP Confidential.4 DNS is an important vector of attack
  • 5. © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. HP Confidential.5 What is DNS? Local DNS Server Root DNS Server .com DNS Server company.com DNS Server Check for Zone Check Cache DNS Traffic Generated by: - Users (e.g. browsing web) - Applications, Servers, etc. Endpoints 8. REPLY: 58.25.88.90 5. REPLY: ask “company.com” 4. QUERY: service.company.com?1. QUERY: service.company.com?
  • 6. © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. HP Confidential.6 1 10 100 1000 10000 100000 1000000 Routers VPN McAfee ePO Active Directory Web Proxy DNS Eventspersecond(logarithmicscale) The Scale of DNS Data 0 20000 40000 60000 80000 100000 120000 140000 Routers VPN McAfee ePO Active Directory Web Proxy DNS Eventspersecond(linearscale) HP IT Pilot: 120,000 DNS events/second
  • 7. © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. HP Confidential.7 What is the Problem? Scale • DNS is one of the most voluminous sources of events in the enterprise Information Granularity • Traditional DNS servers do not log enough detail to detect many kinds of attacks Analysis • Limited analysis tools on top of event collection for DNS
  • 8. © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. HP Confidential.9 End to end handling of DNS events Our Solution Collection Storage Analysis Visualization Filtering out 90-95% DNS traffic - Real-Time - Near-Time/ Historical - Dashboards - Graphs - Timelines …
  • 9. © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. HP Confidential.10 DNS Event Processing: Architecture DNS Server(s) DNS Packet Capture Whitelist Network tapDNS queries and responses ArcSight Logger ArcSight ESM Blacklist HPL Analytical and Visualization Solution Event Logging Correlation & Alerting Blacklist & Whitelist Manager Real-Time Processing Near-time/ Historical Analysis DNS events: Queries & Replies
  • 10. © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. HP Confidential.11 DNS Packet Capturer Security Event Logs Scheduler & Workflows Vertica Anomaly Detectio n Threat Indicators Threat Analytics & Visualization Library HPL Security Analytical & Visualization Network Systems Visual Processin g ArcSight ESM Filtered DNS events Event Downloader & Processor ArcSight Logger Events ESM Alerts Whitelist, RepSM Blacklist Generator Real-time Analysis Near-time/Historical Analysis Hadoop, Autonomy, R Web Server Threat Intelligenc e Anomalies, Threat Findings Anomalies, Threats, Graphs Syslog Server Alerts Introspection
  • 11. © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. HP Confidential.12 Analytic Approaches Detection of Bad Devices and Domain Names: – Large #Queries to Black-listed Domain Names – Anomalous Traffic Patterns – Usage of Machine-generated Domain Names – Timeseries Analysis Success Cases: CryptoLockers, Zeus-Variant, C&C, etc.
  • 12. © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. HP Confidential.13 Analytic Approaches • Mechanisms • ArcSight ESM rules • Vertica SQL Queries • Algorithms & Code • Pipeline/Distribution of Queries and Code • Pilots with HP Labs IT and HP Cybersecurity Defence Center (CDC)
  • 13. © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. HP Confidential.14 Data Visualization Approaches Connected graphs Time series Raw data exploration
  • 14. © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. HP Confidential. Demo …
  • 15. © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. HP Confidential.16
  • 16. © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. HP Confidential.17 Current Status and Next Steps • 2 Pilots: HP Labs IT, HP IT • Lot of HP customers’ interest: 1 pilot to be finalized • Collaborations: HP ESP, TippingPoint/DVLabs, HP ESS, HP IT, HPL IT
  • 17. © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. HP Confidential. Thank you
  • 18. © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. HP Confidential.19 HP HAVEn – Big Data platform HAVEn Social media IT/OT ImagesAudioVideo Transactional dataMobile Search engineEmail Texts Catalog massive volumes of distributed data Hadoop/ HDFS Process and index all information Autonomy IDOL Analyze at extreme scale in real-time Vertica Collect & unify machine data Enterprise Security Powering HP Software + your apps nApps Documents hp.com/haven
  • 19. © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. HP Confidential.20 Competitive Landscape Main HP Competitors in the Security Market already making claims about the following features: • Network Packet Capture capabilities, inclusive of DNS traffic • Back-end advanced Analytics on large amounts of (security) events, complementing real-time event/alert processing

Editor's Notes

  1. Opportunity HP is uniquely positioned to deliver a complete solution around the collection, storage, analysis and visualization of security Big Data, such as DNS logs
  2. NOTE: Show big Event here, before the linear scale (Yolanta and Shane)
  3. . Attacks on DNS servers Denial of Service Cache Poisoning Hijacking/Redirection Code Injection 2. Attacks that leverage DNS to attack third parties Footprinting Reflection & Amplification 3. Attacks that use DNS as infrastructure Communication to malicious servers Fast Flux Domain Name Generation DNS Tunneling
  4. Collection Bypass DNS logs completely. Grab packets directly off the wire using custom hardware Collect the information needed to detect attacks Independent of DNS server vendor Storage Throw out known good data through data analysis Reduce storage requirements by 99% Events stored in ArcSight Logger Split info across multiple CEF fields Analysis Real-time Analysis Correlations & Alerting Utilizing ArcSight Historical Analysis Analytics to detect wide variety of attacks Anomaly Detection Utilizing ArcSight, Vertica, Hadoop, Autonomy, etc. Visualization Novel visualizations to help analysts deal with huge amount of data
  5. 1. Say what the following things are: Whitelists (95% DNS traffic) BlackList (RepSM) Greylists 2. Distinction between Real-time Processing and Near-time/Historical analysis
  6. Lead with innovation: HPL Library of Analytics and Visualization  Threat Indicators, Anomaly Detection, Visual Processing Detect attacks not seen before No need for PhD and data scientists to make sense of outcomes
  7. We have almost 50 metrics we generate to detect bad devices and domain names
  8. Detection of “Bad Devices” and “Malicious Domain Names”: Large amount of traffic towards blacklisted domain names (aggregations) Devices querying machine-generated domain names (bigrams, clustering, ML) Devices’ behavioral anomalies e.g. connecting for first time to dubious domain names Timeseries analysis Mechanisms: Vertica SQL queries Algorithms & Code Pipelines/Distribution of Queries and Code Pilots with HP Labs IT and HP Cybersecurity Defence Center (CDC) Various success cases, including detection of: CryptoLockers , Zeus-variants, malicious domain names, etc
  9. Connected graphs Present a lot of data at the same time Show patterns in DNS queries across multiple clients Time series Show evolution over the time Lets spot abnormally high traffic Raw data exploration Via tables or other charts Allow to spot extra patterns