SlideShare a Scribd company logo
1 of 43
Download to read offline
Raffael Marty, CEO
Big Data Visualization
London
February, 2015
Security. Analytics. Insight.2
• Visualization
• Design Principles
• Dashboards
• SOC Dashboard
• Data Discovery and Exploration
• Data Requirements for Visualization
• Big Data Lake
Overview
Security. Analytics. Insight.3
I am Raffy - I do Viz!
IBM Research
4
Visualization
Security. Analytics. Insight.5
Why Visualization?
the stats ...
http://en.wikipedia.org/wiki/Anscombe%27s_quartet
the data...
Security. Analytics. Insight.6
Why Visualization?
http://en.wikipedia.org/wiki/Anscombe%27s_quartet
Human analyst:
• patterndetection
• remembers context
• fantasticintuition
• canpredict
Security. Analytics. Insight.7
Visualization To …
Present / Communicate Discover / Explore
Design Principles
Security. Analytics. Insight.9
Choosing Visualizations
Objective AudienceData
Security. Analytics. Insight.10
• Objective: Find attackers in the network moving laterally
• Defines data needed (netflow, sflow, …)
• maybe restrict to a network segment
• Audience: security analyst, risk team, …
• Informs how to visualize / present data
For Example - Lateral Movement
Recon Weaponize Deliver Exploit Install C2 Act
Security. Analytics. Insight.11
• Show  comparisons, contrasts,
differences
• Show  causality, mechanism,
explanation, systematic structure.
• Show  multivariate data; that is,
show more than 1 or 2 variables.
by Edward Tufte
Principals of Analytic Design
Security. Analytics. Insight.12
Show Context
42
Security. Analytics. Insight.
42
is just a number
and means nothing without
context
13
Show Context
Security. Analytics. Insight.15
Use Numbers To Highlight Most Important Parts of Data
Numbers
Summaries
Security. Analytics. Insight.16
Additional information about
objects, such as:
• machine
• roles
• criticality
• location
• owner
• …
• user
• roles
• office location
• …
Add Context
source destination
machine and 

user context
machine role
user role
Security. Analytics. Insight.17
Traffic Flow Analysis With Context
Security. Analytics. Insight.18
http://www.scifiinterfaces.com/
• Black background
• Blue or green colors
• Glow
Aesthetics Matter
Security. Analytics. Insight.19
B O R I N G
Security. Analytics. Insight.20
Sexier
Security. Analytics. Insight.21
• Audience, audience, audience!
• Comprehensive Information (enough context)
• Highlight important data
• Use graphics when appropriate
• Good choice of graphics and design
• Aesthetically pleasing
• Enough information to decide if action is necessary
• No scrolling
• Real-time vs. batch? (Refresh-rates)
• Clear organization
Dashboard Design Principles
22
SOC Dashboards
Security. Analytics. Insight.23
Mostly Blank
Security. Analytics. Insight.24
• Disappears too quickly
• Analysts focus is on their own screens
• SOC dashboard just distracts
• Detailed information not legible
• Put the detailed dashboards on the analysts screens!
Dashboards For Discovery
Security. Analytics. Insight.25
• Provide analyst with context
• “What else is going on in the environment right now?”
• Bring Into Focus
• Turn something benign into something interesting
• Disprove
• Turn something interesting into something benign
Use SOC Dashboard For Context
Environment informs detection policies
Security. Analytics. Insight.26
Show Comparisons
Current Measure
week prior
Security. Analytics. Insight.27
• News feed summary (FS ISAC feeds, mailinglists, threat feeds)
• Monitoring twitter or IRC for certain activity / keywords
• Volumes or metrics (e.g., #firewall blocks, #IDS alerts, #failed transactions)
• Top N metrics:
• Top 10 suspicious users
• Top 10 servers connecting outbound
What To Put on Screens
Provide context to individual security alerts
http://raffy.ch/blog/2015/01/15/dashboards-in-the-security-opartions-center-soc/
28
Data Discovery &
Exploration
Security. Analytics. Insight.29
Visualize Me Lots (>1TB) of Data
Security. Analytics. Insight.30
Information Visualization Mantra
Overview Zoom / Filter Details on Demand
Principle by Ben Shneiderman
• summary / aggregation
• data mining
• signal detection (IDS, behavioral, etc.)
Security. Analytics. Insight.31
• Access to data
• Parsed data and data context
• Data architecture for central data access and fast queries
• Application of data mining (how?, what?, scalable, …)
• Visualization tools that support
• Complex visual types (||-coordinates, treemaps, 

heat maps, link graphs)
• Linked views
• Data mining (clustering, …)
• Collaboration, information sharing
• Visual analytics workflow
Visualization Challenges
Big Data Lake
Security. Analytics. Insight.33
• One central location to store all cyber security data
• “Data collected only once and third party software leveraging it”
• Scalability and interoperability
• More than deploying an off the shelf product from a vendor
• Data use influences both data formats and technologies to store the data
• search, analytics, relationships, and distributed processing
• correlation, and statistical summarization
• What to do with Context? Enrich or join?
• Hard problems:
• Parsing: can you re-parse? Common naming scheme!
• Data store capabilities (search, analytics, distributed processing, etc.)
• Access to data: SQL (even in Hadoop context), how can products access the data?
The Big Data Lake
Security. Analytics. Insight.34
Federated Data Access
SIEM
dispatcher
SIEM 

connector
SIEM console
Prod A
AD / LDAP
HR
…
IDS
FW
Prod B
DBs
Data Lake
Caveats:
• Dispatcher?
• Standard access to dispatcher /

products enabled
• Data lake technology?
SNMP
Security. Analytics. Insight.35
Multiple Data Stores
raw logs
key-value
structured
real-time

processing
(un)-structured data
context
SQL
s
t
o
r
a
g
e
stats
index
queue
distributed

processing
a
c
c
e
s
s
graph
Caveat:
• Need multiple types of 

data stores
Security. Analytics. Insight.36
Technologies (Example)
raw logs
key-value
(Cassandra)
columnar
(parquet)
real-time

processing
(Spark)
(un)-structured data
context
SQL
(Impala,
SparkSQL)
H
D
F
S
aggregates
index
(ES)
queue
(Kafka)
distributed

processing
(Spark)
a
c
c
e
s
s
graph
(GraphX)
Caveat:
• No out of the box
solution available
Security. Analytics. Insight.37
SIEM Integration - Log Management First
SIEM
columnar
or
search engine

or
log management
processing
SIEM 

connector
raw logs
SIEM console
SQL or search

interface
processing
filtering
H
D
F
S
e.g., PIG parsing
Security. Analytics. Insight.38
Simple SIEM Integration
raw, csv, json
flume
log data
SQL
(Impala,
with SerDe)
H
D
F
S
SIEM 

connector
SIEM
Requirement:
• SIEM connector to forward text-
based data to Flume.
SQL interface
Tableau, etc.
SIEM console
Security. Analytics. Insight.39
SIEM Integration - Advanced
SIEM
columnar
(parquet)
processing
syslog data
SQL
(Impala,
SparkSQL)
H
D
F
S
index
(ES)
queue
(Kafka)
a
c
c
e
s
s
other data
sources
SIEM 

connector
raw logs
SIEM console
SQL and search 

interface
Tableau, Kibana, etc.
requires parsing and
formatting in a SIEM
readable format (e.g., CEF)
Security. Analytics. Insight.40
What I am Working On
Data Stores Analytics Forensics Models Admin
10.9.79.109 --> 3.16.204.150
10.8.24.80 --> 192.168.148.193
10.8.50.85 --> 192.168.148.193
10.8.48.128 --> 192.168.148.193
10.9.79.6 --> 192.168.148.193
10.9.79.6
10.8.48.128
80
53
8.8.8.8
127.0.0.1
Anomalies
Decomposition
Data
Seasonal
Trend
Anomaly Details
“Hunt” ExplainVisual Search
• Big data backend
• Own visualization engine (Web-based)
• Visualization workflows
Security. Analytics. Insight.41
BlackHat Workshop
Visual Analytics -
Delivering Actionable Security
Intelligence
August 1-6 2015, Las Vegas, USA
big data | analytics | visualization
Security. Analytics. Insight.42
http://secviz.org
List: secviz.org/mailinglist
Twitter: @secviz
Share, discuss, challenge, and learn about security visualization.
Security Visualization Community
Security. Analytics. Insight.
raffael.marty@pixlcloud.com
http://slideshare.net/zrlram
http://secviz.org and @secviz
Further resources:

More Related Content

What's hot

Introduction to Data Analytics
Introduction to Data AnalyticsIntroduction to Data Analytics
Introduction to Data AnalyticsUtkarsh Sharma
 
Introduction to Data Visualization
Introduction to Data Visualization Introduction to Data Visualization
Introduction to Data Visualization Ana Jofre
 
Introduction on Data Science
Introduction on Data ScienceIntroduction on Data Science
Introduction on Data ScienceEdureka!
 
Big Data Architecture
Big Data ArchitectureBig Data Architecture
Big Data ArchitectureGuido Schmutz
 
Introduction to Data Engineering
Introduction to Data EngineeringIntroduction to Data Engineering
Introduction to Data EngineeringHadi Fadlallah
 
Data Analytics For Beginners | Introduction To Data Analytics | Data Analytic...
Data Analytics For Beginners | Introduction To Data Analytics | Data Analytic...Data Analytics For Beginners | Introduction To Data Analytics | Data Analytic...
Data Analytics For Beginners | Introduction To Data Analytics | Data Analytic...Edureka!
 
Top 8 Data Science Tools | Open Source Tools for Data Scientists | Edureka
Top 8 Data Science Tools | Open Source Tools for Data Scientists | EdurekaTop 8 Data Science Tools | Open Source Tools for Data Scientists | Edureka
Top 8 Data Science Tools | Open Source Tools for Data Scientists | EdurekaEdureka!
 
Data science & data scientist
Data science & data scientistData science & data scientist
Data science & data scientistVijayMohan Vasu
 
Exploratory data analysis
Exploratory data analysis Exploratory data analysis
Exploratory data analysis Peter Reimann
 
Big Data Characteristics And Process PowerPoint Presentation Slides
Big Data Characteristics And Process PowerPoint Presentation SlidesBig Data Characteristics And Process PowerPoint Presentation Slides
Big Data Characteristics And Process PowerPoint Presentation SlidesSlideTeam
 
Best Practices in DataOps: How to Create Agile, Automated Data Pipelines
Best Practices in DataOps: How to Create Agile, Automated Data PipelinesBest Practices in DataOps: How to Create Agile, Automated Data Pipelines
Best Practices in DataOps: How to Create Agile, Automated Data PipelinesEric Kavanagh
 
Big Data Analytics
Big Data AnalyticsBig Data Analytics
Big Data AnalyticsRohithND
 

What's hot (20)

Introduction to Data Analytics
Introduction to Data AnalyticsIntroduction to Data Analytics
Introduction to Data Analytics
 
Introduction to Data Visualization
Introduction to Data Visualization Introduction to Data Visualization
Introduction to Data Visualization
 
Introduction to data science
Introduction to data scienceIntroduction to data science
Introduction to data science
 
Data lake ppt
Data lake pptData lake ppt
Data lake ppt
 
Data Engineering Basics
Data Engineering BasicsData Engineering Basics
Data Engineering Basics
 
Introduction on Data Science
Introduction on Data ScienceIntroduction on Data Science
Introduction on Data Science
 
Data science
Data scienceData science
Data science
 
Big Data Architecture
Big Data ArchitectureBig Data Architecture
Big Data Architecture
 
Introduction to Data Engineering
Introduction to Data EngineeringIntroduction to Data Engineering
Introduction to Data Engineering
 
Data Analytics For Beginners | Introduction To Data Analytics | Data Analytic...
Data Analytics For Beginners | Introduction To Data Analytics | Data Analytic...Data Analytics For Beginners | Introduction To Data Analytics | Data Analytic...
Data Analytics For Beginners | Introduction To Data Analytics | Data Analytic...
 
Data science
Data scienceData science
Data science
 
Top 8 Data Science Tools | Open Source Tools for Data Scientists | Edureka
Top 8 Data Science Tools | Open Source Tools for Data Scientists | EdurekaTop 8 Data Science Tools | Open Source Tools for Data Scientists | Edureka
Top 8 Data Science Tools | Open Source Tools for Data Scientists | Edureka
 
Data science Big Data
Data science Big DataData science Big Data
Data science Big Data
 
Data science & data scientist
Data science & data scientistData science & data scientist
Data science & data scientist
 
Exploratory data analysis
Exploratory data analysis Exploratory data analysis
Exploratory data analysis
 
Big Data Characteristics And Process PowerPoint Presentation Slides
Big Data Characteristics And Process PowerPoint Presentation SlidesBig Data Characteristics And Process PowerPoint Presentation Slides
Big Data Characteristics And Process PowerPoint Presentation Slides
 
Best Practices in DataOps: How to Create Agile, Automated Data Pipelines
Best Practices in DataOps: How to Create Agile, Automated Data PipelinesBest Practices in DataOps: How to Create Agile, Automated Data Pipelines
Best Practices in DataOps: How to Create Agile, Automated Data Pipelines
 
Data Science
Data ScienceData Science
Data Science
 
Big Data Analytics
Big Data AnalyticsBig Data Analytics
Big Data Analytics
 
Data Mesh
Data MeshData Mesh
Data Mesh
 

Similar to Raffael Marty Big Data Visualization London Presentation

Visualization in the Age of Big Data
Visualization in the Age of Big DataVisualization in the Age of Big Data
Visualization in the Age of Big DataRaffael Marty
 
Building Your Application Security Data Hub - OWASP AppSecUSA
Building Your Application Security Data Hub - OWASP AppSecUSABuilding Your Application Security Data Hub - OWASP AppSecUSA
Building Your Application Security Data Hub - OWASP AppSecUSADenim Group
 
DevSecCon London 2017: when good containers go bad by Tim Mackey
DevSecCon London 2017: when good containers go bad by Tim MackeyDevSecCon London 2017: when good containers go bad by Tim Mackey
DevSecCon London 2017: when good containers go bad by Tim MackeyDevSecCon
 
PLNOG19 - Gaweł Mikołajczyk & Michał Garcarz - SOC, studium ciężkich przypadków
PLNOG19 - Gaweł Mikołajczyk & Michał Garcarz - SOC, studium ciężkich przypadkówPLNOG19 - Gaweł Mikołajczyk & Michał Garcarz - SOC, studium ciężkich przypadków
PLNOG19 - Gaweł Mikołajczyk & Michał Garcarz - SOC, studium ciężkich przypadkówPROIDEA
 
Mining software vulns in SCCM / NIST's NVD
Mining software vulns in SCCM / NIST's NVDMining software vulns in SCCM / NIST's NVD
Mining software vulns in SCCM / NIST's NVDLoren Gordon
 
Continuum Analytics and Python
Continuum Analytics and PythonContinuum Analytics and Python
Continuum Analytics and PythonTravis Oliphant
 
Soc analyst course content v3
Soc analyst course content v3Soc analyst course content v3
Soc analyst course content v3ShivamSharma909
 
Soc analyst course content
Soc analyst course contentSoc analyst course content
Soc analyst course contentShivamSharma909
 
Visualization for Security
Visualization for SecurityVisualization for Security
Visualization for SecurityRaffael Marty
 
Workshop: Big Data Visualization for Security
Workshop: Big Data Visualization for SecurityWorkshop: Big Data Visualization for Security
Workshop: Big Data Visualization for SecurityRaffael Marty
 
Expose Yourself Without Insecurity: Cloud Breach Patterns
Expose Yourself Without Insecurity: Cloud Breach PatternsExpose Yourself Without Insecurity: Cloud Breach Patterns
Expose Yourself Without Insecurity: Cloud Breach PatternsRob Ragan
 
Accelerating Data Lakes and Streams with Real-time Analytics
Accelerating Data Lakes and Streams with Real-time AnalyticsAccelerating Data Lakes and Streams with Real-time Analytics
Accelerating Data Lakes and Streams with Real-time AnalyticsArcadia Data
 
WSO2Con ASIA 2016: Patterns for Deploying Analytics in the Real World
WSO2Con ASIA 2016: Patterns for Deploying Analytics in the Real WorldWSO2Con ASIA 2016: Patterns for Deploying Analytics in the Real World
WSO2Con ASIA 2016: Patterns for Deploying Analytics in the Real WorldWSO2
 
Deploying, Managing, and Leveraging Honeypots in the Enterprise using Open So...
Deploying, Managing, and Leveraging Honeypots in the Enterprise using Open So...Deploying, Managing, and Leveraging Honeypots in the Enterprise using Open So...
Deploying, Managing, and Leveraging Honeypots in the Enterprise using Open So...Jason Trost
 
DevOps for Data Engineers - Automate Your Data Science Pipeline with Ansible,...
DevOps for Data Engineers - Automate Your Data Science Pipeline with Ansible,...DevOps for Data Engineers - Automate Your Data Science Pipeline with Ansible,...
DevOps for Data Engineers - Automate Your Data Science Pipeline with Ansible,...Mihai Criveti
 
Cloud Security at Netflix
Cloud Security at NetflixCloud Security at Netflix
Cloud Security at NetflixJason Chan
 
Threat_Modelling.pdf
Threat_Modelling.pdfThreat_Modelling.pdf
Threat_Modelling.pdfMarlboroAbyad
 
20160000 Cloud Discovery Event - Cloud Access Security Brokers
20160000 Cloud Discovery Event - Cloud Access Security Brokers20160000 Cloud Discovery Event - Cloud Access Security Brokers
20160000 Cloud Discovery Event - Cloud Access Security BrokersRobin Vermeirsch
 
Open Source Defense for Edge 2017
Open Source Defense for Edge 2017Open Source Defense for Edge 2017
Open Source Defense for Edge 2017Adrian Sanabria
 

Similar to Raffael Marty Big Data Visualization London Presentation (20)

Visualization in the Age of Big Data
Visualization in the Age of Big DataVisualization in the Age of Big Data
Visualization in the Age of Big Data
 
Building Your Application Security Data Hub - OWASP AppSecUSA
Building Your Application Security Data Hub - OWASP AppSecUSABuilding Your Application Security Data Hub - OWASP AppSecUSA
Building Your Application Security Data Hub - OWASP AppSecUSA
 
DevSecCon London 2017: when good containers go bad by Tim Mackey
DevSecCon London 2017: when good containers go bad by Tim MackeyDevSecCon London 2017: when good containers go bad by Tim Mackey
DevSecCon London 2017: when good containers go bad by Tim Mackey
 
PLNOG19 - Gaweł Mikołajczyk & Michał Garcarz - SOC, studium ciężkich przypadków
PLNOG19 - Gaweł Mikołajczyk & Michał Garcarz - SOC, studium ciężkich przypadkówPLNOG19 - Gaweł Mikołajczyk & Michał Garcarz - SOC, studium ciężkich przypadków
PLNOG19 - Gaweł Mikołajczyk & Michał Garcarz - SOC, studium ciężkich przypadków
 
Mining software vulns in SCCM / NIST's NVD
Mining software vulns in SCCM / NIST's NVDMining software vulns in SCCM / NIST's NVD
Mining software vulns in SCCM / NIST's NVD
 
Continuum Analytics and Python
Continuum Analytics and PythonContinuum Analytics and Python
Continuum Analytics and Python
 
Soc analyst course content v3
Soc analyst course content v3Soc analyst course content v3
Soc analyst course content v3
 
Soc analyst course content
Soc analyst course contentSoc analyst course content
Soc analyst course content
 
Visualization for Security
Visualization for SecurityVisualization for Security
Visualization for Security
 
Workshop: Big Data Visualization for Security
Workshop: Big Data Visualization for SecurityWorkshop: Big Data Visualization for Security
Workshop: Big Data Visualization for Security
 
Expose Yourself Without Insecurity: Cloud Breach Patterns
Expose Yourself Without Insecurity: Cloud Breach PatternsExpose Yourself Without Insecurity: Cloud Breach Patterns
Expose Yourself Without Insecurity: Cloud Breach Patterns
 
Accelerating Data Lakes and Streams with Real-time Analytics
Accelerating Data Lakes and Streams with Real-time AnalyticsAccelerating Data Lakes and Streams with Real-time Analytics
Accelerating Data Lakes and Streams with Real-time Analytics
 
WSO2Con ASIA 2016: Patterns for Deploying Analytics in the Real World
WSO2Con ASIA 2016: Patterns for Deploying Analytics in the Real WorldWSO2Con ASIA 2016: Patterns for Deploying Analytics in the Real World
WSO2Con ASIA 2016: Patterns for Deploying Analytics in the Real World
 
Deploying, Managing, and Leveraging Honeypots in the Enterprise using Open So...
Deploying, Managing, and Leveraging Honeypots in the Enterprise using Open So...Deploying, Managing, and Leveraging Honeypots in the Enterprise using Open So...
Deploying, Managing, and Leveraging Honeypots in the Enterprise using Open So...
 
DevOps for Data Engineers - Automate Your Data Science Pipeline with Ansible,...
DevOps for Data Engineers - Automate Your Data Science Pipeline with Ansible,...DevOps for Data Engineers - Automate Your Data Science Pipeline with Ansible,...
DevOps for Data Engineers - Automate Your Data Science Pipeline with Ansible,...
 
Cloud Security at Netflix
Cloud Security at NetflixCloud Security at Netflix
Cloud Security at Netflix
 
Threat_Modelling.pdf
Threat_Modelling.pdfThreat_Modelling.pdf
Threat_Modelling.pdf
 
20160000 Cloud Discovery Event - Cloud Access Security Brokers
20160000 Cloud Discovery Event - Cloud Access Security Brokers20160000 Cloud Discovery Event - Cloud Access Security Brokers
20160000 Cloud Discovery Event - Cloud Access Security Brokers
 
Security for developers
Security for developersSecurity for developers
Security for developers
 
Open Source Defense for Edge 2017
Open Source Defense for Edge 2017Open Source Defense for Edge 2017
Open Source Defense for Edge 2017
 

More from Raffael Marty

Exploring the Defender's Advantage
Exploring the Defender's AdvantageExploring the Defender's Advantage
Exploring the Defender's AdvantageRaffael Marty
 
Extended Detection and Response (XDR) An Overhyped Product Category With Ulti...
Extended Detection and Response (XDR)An Overhyped Product Category With Ulti...Extended Detection and Response (XDR)An Overhyped Product Category With Ulti...
Extended Detection and Response (XDR) An Overhyped Product Category With Ulti...Raffael Marty
 
How To Drive Value with Security Data
How To Drive Value with Security DataHow To Drive Value with Security Data
How To Drive Value with Security DataRaffael Marty
 
Cyber Security Beyond 2020 – Will We Learn From Our Mistakes?
Cyber Security Beyond 2020 – Will We Learn From Our Mistakes?Cyber Security Beyond 2020 – Will We Learn From Our Mistakes?
Cyber Security Beyond 2020 – Will We Learn From Our Mistakes?Raffael Marty
 
Artificial Intelligence – Time Bomb or The Promised Land?
Artificial Intelligence – Time Bomb or The Promised Land?Artificial Intelligence – Time Bomb or The Promised Land?
Artificial Intelligence – Time Bomb or The Promised Land?Raffael Marty
 
Understanding the "Intelligence" in AI
Understanding the "Intelligence" in AIUnderstanding the "Intelligence" in AI
Understanding the "Intelligence" in AIRaffael Marty
 
AI & ML in Cyber Security - Why Algorithms are Dangerous
AI & ML in Cyber Security - Why Algorithms are DangerousAI & ML in Cyber Security - Why Algorithms are Dangerous
AI & ML in Cyber Security - Why Algorithms are DangerousRaffael Marty
 
AI & ML in Cyber Security - Why Algorithms Are Dangerous
AI & ML in Cyber Security - Why Algorithms Are DangerousAI & ML in Cyber Security - Why Algorithms Are Dangerous
AI & ML in Cyber Security - Why Algorithms Are DangerousRaffael Marty
 
Delivering Security Insights with Data Analytics and Visualization
Delivering Security Insights with Data Analytics and VisualizationDelivering Security Insights with Data Analytics and Visualization
Delivering Security Insights with Data Analytics and VisualizationRaffael Marty
 
AI & ML in Cyber Security - Welcome Back to 1999 - Security Hasn't Changed
AI & ML in Cyber Security - Welcome Back to 1999 - Security Hasn't ChangedAI & ML in Cyber Security - Welcome Back to 1999 - Security Hasn't Changed
AI & ML in Cyber Security - Welcome Back to 1999 - Security Hasn't ChangedRaffael Marty
 
Security Insights at Scale
Security Insights at ScaleSecurity Insights at Scale
Security Insights at ScaleRaffael Marty
 
Creating Your Own Threat Intel Through Hunting & Visualization
Creating Your Own Threat Intel Through Hunting & VisualizationCreating Your Own Threat Intel Through Hunting & Visualization
Creating Your Own Threat Intel Through Hunting & VisualizationRaffael Marty
 
Creating Your Own Threat Intel Through Hunting & Visualization
Creating Your Own Threat Intel Through Hunting & VisualizationCreating Your Own Threat Intel Through Hunting & Visualization
Creating Your Own Threat Intel Through Hunting & VisualizationRaffael Marty
 
The Heatmap
 - Why is Security Visualization so Hard?
The Heatmap
 - Why is Security Visualization so Hard?The Heatmap
 - Why is Security Visualization so Hard?
The Heatmap
 - Why is Security Visualization so Hard?Raffael Marty
 
The Heatmap
 - Why is Security Visualization so Hard?
The Heatmap
 - Why is Security Visualization so Hard?The Heatmap
 - Why is Security Visualization so Hard?
The Heatmap
 - Why is Security Visualization so Hard?Raffael Marty
 
DAVIX - Data Analysis and Visualization Linux
DAVIX - Data Analysis and Visualization LinuxDAVIX - Data Analysis and Visualization Linux
DAVIX - Data Analysis and Visualization LinuxRaffael Marty
 
Cloud - Security - Big Data
Cloud - Security - Big DataCloud - Security - Big Data
Cloud - Security - Big DataRaffael Marty
 
Cyber Security – How Visual Analytics Unlock Insight
Cyber Security – How Visual Analytics Unlock InsightCyber Security – How Visual Analytics Unlock Insight
Cyber Security – How Visual Analytics Unlock InsightRaffael Marty
 

More from Raffael Marty (20)

Exploring the Defender's Advantage
Exploring the Defender's AdvantageExploring the Defender's Advantage
Exploring the Defender's Advantage
 
Extended Detection and Response (XDR) An Overhyped Product Category With Ulti...
Extended Detection and Response (XDR)An Overhyped Product Category With Ulti...Extended Detection and Response (XDR)An Overhyped Product Category With Ulti...
Extended Detection and Response (XDR) An Overhyped Product Category With Ulti...
 
How To Drive Value with Security Data
How To Drive Value with Security DataHow To Drive Value with Security Data
How To Drive Value with Security Data
 
Cyber Security Beyond 2020 – Will We Learn From Our Mistakes?
Cyber Security Beyond 2020 – Will We Learn From Our Mistakes?Cyber Security Beyond 2020 – Will We Learn From Our Mistakes?
Cyber Security Beyond 2020 – Will We Learn From Our Mistakes?
 
Artificial Intelligence – Time Bomb or The Promised Land?
Artificial Intelligence – Time Bomb or The Promised Land?Artificial Intelligence – Time Bomb or The Promised Land?
Artificial Intelligence – Time Bomb or The Promised Land?
 
Understanding the "Intelligence" in AI
Understanding the "Intelligence" in AIUnderstanding the "Intelligence" in AI
Understanding the "Intelligence" in AI
 
Security Chat 5.0
Security Chat 5.0Security Chat 5.0
Security Chat 5.0
 
AI & ML in Cyber Security - Why Algorithms are Dangerous
AI & ML in Cyber Security - Why Algorithms are DangerousAI & ML in Cyber Security - Why Algorithms are Dangerous
AI & ML in Cyber Security - Why Algorithms are Dangerous
 
AI & ML in Cyber Security - Why Algorithms Are Dangerous
AI & ML in Cyber Security - Why Algorithms Are DangerousAI & ML in Cyber Security - Why Algorithms Are Dangerous
AI & ML in Cyber Security - Why Algorithms Are Dangerous
 
Delivering Security Insights with Data Analytics and Visualization
Delivering Security Insights with Data Analytics and VisualizationDelivering Security Insights with Data Analytics and Visualization
Delivering Security Insights with Data Analytics and Visualization
 
AI & ML in Cyber Security - Welcome Back to 1999 - Security Hasn't Changed
AI & ML in Cyber Security - Welcome Back to 1999 - Security Hasn't ChangedAI & ML in Cyber Security - Welcome Back to 1999 - Security Hasn't Changed
AI & ML in Cyber Security - Welcome Back to 1999 - Security Hasn't Changed
 
Security Insights at Scale
Security Insights at ScaleSecurity Insights at Scale
Security Insights at Scale
 
Creating Your Own Threat Intel Through Hunting & Visualization
Creating Your Own Threat Intel Through Hunting & VisualizationCreating Your Own Threat Intel Through Hunting & Visualization
Creating Your Own Threat Intel Through Hunting & Visualization
 
Creating Your Own Threat Intel Through Hunting & Visualization
Creating Your Own Threat Intel Through Hunting & VisualizationCreating Your Own Threat Intel Through Hunting & Visualization
Creating Your Own Threat Intel Through Hunting & Visualization
 
The Heatmap
 - Why is Security Visualization so Hard?
The Heatmap
 - Why is Security Visualization so Hard?The Heatmap
 - Why is Security Visualization so Hard?
The Heatmap
 - Why is Security Visualization so Hard?
 
The Heatmap
 - Why is Security Visualization so Hard?
The Heatmap
 - Why is Security Visualization so Hard?The Heatmap
 - Why is Security Visualization so Hard?
The Heatmap
 - Why is Security Visualization so Hard?
 
DAVIX - Data Analysis and Visualization Linux
DAVIX - Data Analysis and Visualization LinuxDAVIX - Data Analysis and Visualization Linux
DAVIX - Data Analysis and Visualization Linux
 
Cloud - Security - Big Data
Cloud - Security - Big DataCloud - Security - Big Data
Cloud - Security - Big Data
 
Cyber Security – How Visual Analytics Unlock Insight
Cyber Security – How Visual Analytics Unlock InsightCyber Security – How Visual Analytics Unlock Insight
Cyber Security – How Visual Analytics Unlock Insight
 
AfterGlow
AfterGlowAfterGlow
AfterGlow
 

Recently uploaded

INTERNSHIP ON PURBASHA COMPOSITE TEX LTD
INTERNSHIP ON PURBASHA COMPOSITE TEX LTDINTERNSHIP ON PURBASHA COMPOSITE TEX LTD
INTERNSHIP ON PURBASHA COMPOSITE TEX LTDRafezzaman
 
Identifying Appropriate Test Statistics Involving Population Mean
Identifying Appropriate Test Statistics Involving Population MeanIdentifying Appropriate Test Statistics Involving Population Mean
Identifying Appropriate Test Statistics Involving Population MeanMYRABACSAFRA2
 
Predicting Salary Using Data Science: A Comprehensive Analysis.pdf
Predicting Salary Using Data Science: A Comprehensive Analysis.pdfPredicting Salary Using Data Science: A Comprehensive Analysis.pdf
Predicting Salary Using Data Science: A Comprehensive Analysis.pdfBoston Institute of Analytics
 
DBA Basics: Getting Started with Performance Tuning.pdf
DBA Basics: Getting Started with Performance Tuning.pdfDBA Basics: Getting Started with Performance Tuning.pdf
DBA Basics: Getting Started with Performance Tuning.pdfJohn Sterrett
 
专业一比一美国俄亥俄大学毕业证成绩单pdf电子版制作修改
专业一比一美国俄亥俄大学毕业证成绩单pdf电子版制作修改专业一比一美国俄亥俄大学毕业证成绩单pdf电子版制作修改
专业一比一美国俄亥俄大学毕业证成绩单pdf电子版制作修改yuu sss
 
办理(Vancouver毕业证书)加拿大温哥华岛大学毕业证成绩单原版一比一
办理(Vancouver毕业证书)加拿大温哥华岛大学毕业证成绩单原版一比一办理(Vancouver毕业证书)加拿大温哥华岛大学毕业证成绩单原版一比一
办理(Vancouver毕业证书)加拿大温哥华岛大学毕业证成绩单原版一比一F La
 
Business Analytics using Microsoft Excel
Business Analytics using Microsoft ExcelBusiness Analytics using Microsoft Excel
Business Analytics using Microsoft Excelysmaelreyes
 
ASML's Taxonomy Adventure by Daniel Canter
ASML's Taxonomy Adventure by Daniel CanterASML's Taxonomy Adventure by Daniel Canter
ASML's Taxonomy Adventure by Daniel Cantervoginip
 
1:1定制(UQ毕业证)昆士兰大学毕业证成绩单修改留信学历认证原版一模一样
1:1定制(UQ毕业证)昆士兰大学毕业证成绩单修改留信学历认证原版一模一样1:1定制(UQ毕业证)昆士兰大学毕业证成绩单修改留信学历认证原版一模一样
1:1定制(UQ毕业证)昆士兰大学毕业证成绩单修改留信学历认证原版一模一样vhwb25kk
 
How we prevented account sharing with MFA
How we prevented account sharing with MFAHow we prevented account sharing with MFA
How we prevented account sharing with MFAAndrei Kaleshka
 
IMA MSN - Medical Students Network (2).pptx
IMA MSN - Medical Students Network (2).pptxIMA MSN - Medical Students Network (2).pptx
IMA MSN - Medical Students Network (2).pptxdolaknnilon
 
Learn How Data Science Changes Our World
Learn How Data Science Changes Our WorldLearn How Data Science Changes Our World
Learn How Data Science Changes Our WorldEduminds Learning
 
Student profile product demonstration on grades, ability, well-being and mind...
Student profile product demonstration on grades, ability, well-being and mind...Student profile product demonstration on grades, ability, well-being and mind...
Student profile product demonstration on grades, ability, well-being and mind...Seán Kennedy
 
Heart Disease Classification Report: A Data Analysis Project
Heart Disease Classification Report: A Data Analysis ProjectHeart Disease Classification Report: A Data Analysis Project
Heart Disease Classification Report: A Data Analysis ProjectBoston Institute of Analytics
 
原版1:1定制南十字星大学毕业证(SCU毕业证)#文凭成绩单#真实留信学历认证永久存档
原版1:1定制南十字星大学毕业证(SCU毕业证)#文凭成绩单#真实留信学历认证永久存档原版1:1定制南十字星大学毕业证(SCU毕业证)#文凭成绩单#真实留信学历认证永久存档
原版1:1定制南十字星大学毕业证(SCU毕业证)#文凭成绩单#真实留信学历认证永久存档208367051
 
Decoding the Heart: Student Presentation on Heart Attack Prediction with Data...
Decoding the Heart: Student Presentation on Heart Attack Prediction with Data...Decoding the Heart: Student Presentation on Heart Attack Prediction with Data...
Decoding the Heart: Student Presentation on Heart Attack Prediction with Data...Boston Institute of Analytics
 
9711147426✨Call In girls Gurgaon Sector 31. SCO 25 escort service
9711147426✨Call In girls Gurgaon Sector 31. SCO 25 escort service9711147426✨Call In girls Gurgaon Sector 31. SCO 25 escort service
9711147426✨Call In girls Gurgaon Sector 31. SCO 25 escort servicejennyeacort
 
科罗拉多大学波尔得分校毕业证学位证成绩单-可办理
科罗拉多大学波尔得分校毕业证学位证成绩单-可办理科罗拉多大学波尔得分校毕业证学位证成绩单-可办理
科罗拉多大学波尔得分校毕业证学位证成绩单-可办理e4aez8ss
 
Real-Time AI Streaming - AI Max Princeton
Real-Time AI  Streaming - AI Max PrincetonReal-Time AI  Streaming - AI Max Princeton
Real-Time AI Streaming - AI Max PrincetonTimothy Spann
 
NLP Data Science Project Presentation:Predicting Heart Disease with NLP Data ...
NLP Data Science Project Presentation:Predicting Heart Disease with NLP Data ...NLP Data Science Project Presentation:Predicting Heart Disease with NLP Data ...
NLP Data Science Project Presentation:Predicting Heart Disease with NLP Data ...Boston Institute of Analytics
 

Recently uploaded (20)

INTERNSHIP ON PURBASHA COMPOSITE TEX LTD
INTERNSHIP ON PURBASHA COMPOSITE TEX LTDINTERNSHIP ON PURBASHA COMPOSITE TEX LTD
INTERNSHIP ON PURBASHA COMPOSITE TEX LTD
 
Identifying Appropriate Test Statistics Involving Population Mean
Identifying Appropriate Test Statistics Involving Population MeanIdentifying Appropriate Test Statistics Involving Population Mean
Identifying Appropriate Test Statistics Involving Population Mean
 
Predicting Salary Using Data Science: A Comprehensive Analysis.pdf
Predicting Salary Using Data Science: A Comprehensive Analysis.pdfPredicting Salary Using Data Science: A Comprehensive Analysis.pdf
Predicting Salary Using Data Science: A Comprehensive Analysis.pdf
 
DBA Basics: Getting Started with Performance Tuning.pdf
DBA Basics: Getting Started with Performance Tuning.pdfDBA Basics: Getting Started with Performance Tuning.pdf
DBA Basics: Getting Started with Performance Tuning.pdf
 
专业一比一美国俄亥俄大学毕业证成绩单pdf电子版制作修改
专业一比一美国俄亥俄大学毕业证成绩单pdf电子版制作修改专业一比一美国俄亥俄大学毕业证成绩单pdf电子版制作修改
专业一比一美国俄亥俄大学毕业证成绩单pdf电子版制作修改
 
办理(Vancouver毕业证书)加拿大温哥华岛大学毕业证成绩单原版一比一
办理(Vancouver毕业证书)加拿大温哥华岛大学毕业证成绩单原版一比一办理(Vancouver毕业证书)加拿大温哥华岛大学毕业证成绩单原版一比一
办理(Vancouver毕业证书)加拿大温哥华岛大学毕业证成绩单原版一比一
 
Business Analytics using Microsoft Excel
Business Analytics using Microsoft ExcelBusiness Analytics using Microsoft Excel
Business Analytics using Microsoft Excel
 
ASML's Taxonomy Adventure by Daniel Canter
ASML's Taxonomy Adventure by Daniel CanterASML's Taxonomy Adventure by Daniel Canter
ASML's Taxonomy Adventure by Daniel Canter
 
1:1定制(UQ毕业证)昆士兰大学毕业证成绩单修改留信学历认证原版一模一样
1:1定制(UQ毕业证)昆士兰大学毕业证成绩单修改留信学历认证原版一模一样1:1定制(UQ毕业证)昆士兰大学毕业证成绩单修改留信学历认证原版一模一样
1:1定制(UQ毕业证)昆士兰大学毕业证成绩单修改留信学历认证原版一模一样
 
How we prevented account sharing with MFA
How we prevented account sharing with MFAHow we prevented account sharing with MFA
How we prevented account sharing with MFA
 
IMA MSN - Medical Students Network (2).pptx
IMA MSN - Medical Students Network (2).pptxIMA MSN - Medical Students Network (2).pptx
IMA MSN - Medical Students Network (2).pptx
 
Learn How Data Science Changes Our World
Learn How Data Science Changes Our WorldLearn How Data Science Changes Our World
Learn How Data Science Changes Our World
 
Student profile product demonstration on grades, ability, well-being and mind...
Student profile product demonstration on grades, ability, well-being and mind...Student profile product demonstration on grades, ability, well-being and mind...
Student profile product demonstration on grades, ability, well-being and mind...
 
Heart Disease Classification Report: A Data Analysis Project
Heart Disease Classification Report: A Data Analysis ProjectHeart Disease Classification Report: A Data Analysis Project
Heart Disease Classification Report: A Data Analysis Project
 
原版1:1定制南十字星大学毕业证(SCU毕业证)#文凭成绩单#真实留信学历认证永久存档
原版1:1定制南十字星大学毕业证(SCU毕业证)#文凭成绩单#真实留信学历认证永久存档原版1:1定制南十字星大学毕业证(SCU毕业证)#文凭成绩单#真实留信学历认证永久存档
原版1:1定制南十字星大学毕业证(SCU毕业证)#文凭成绩单#真实留信学历认证永久存档
 
Decoding the Heart: Student Presentation on Heart Attack Prediction with Data...
Decoding the Heart: Student Presentation on Heart Attack Prediction with Data...Decoding the Heart: Student Presentation on Heart Attack Prediction with Data...
Decoding the Heart: Student Presentation on Heart Attack Prediction with Data...
 
9711147426✨Call In girls Gurgaon Sector 31. SCO 25 escort service
9711147426✨Call In girls Gurgaon Sector 31. SCO 25 escort service9711147426✨Call In girls Gurgaon Sector 31. SCO 25 escort service
9711147426✨Call In girls Gurgaon Sector 31. SCO 25 escort service
 
科罗拉多大学波尔得分校毕业证学位证成绩单-可办理
科罗拉多大学波尔得分校毕业证学位证成绩单-可办理科罗拉多大学波尔得分校毕业证学位证成绩单-可办理
科罗拉多大学波尔得分校毕业证学位证成绩单-可办理
 
Real-Time AI Streaming - AI Max Princeton
Real-Time AI  Streaming - AI Max PrincetonReal-Time AI  Streaming - AI Max Princeton
Real-Time AI Streaming - AI Max Princeton
 
NLP Data Science Project Presentation:Predicting Heart Disease with NLP Data ...
NLP Data Science Project Presentation:Predicting Heart Disease with NLP Data ...NLP Data Science Project Presentation:Predicting Heart Disease with NLP Data ...
NLP Data Science Project Presentation:Predicting Heart Disease with NLP Data ...
 

Raffael Marty Big Data Visualization London Presentation

  • 1. Raffael Marty, CEO Big Data Visualization London February, 2015
  • 2. Security. Analytics. Insight.2 • Visualization • Design Principles • Dashboards • SOC Dashboard • Data Discovery and Exploration • Data Requirements for Visualization • Big Data Lake Overview
  • 3. Security. Analytics. Insight.3 I am Raffy - I do Viz! IBM Research
  • 5. Security. Analytics. Insight.5 Why Visualization? the stats ... http://en.wikipedia.org/wiki/Anscombe%27s_quartet the data...
  • 6. Security. Analytics. Insight.6 Why Visualization? http://en.wikipedia.org/wiki/Anscombe%27s_quartet Human analyst: • patterndetection • remembers context • fantasticintuition • canpredict
  • 7. Security. Analytics. Insight.7 Visualization To … Present / Communicate Discover / Explore
  • 9. Security. Analytics. Insight.9 Choosing Visualizations Objective AudienceData
  • 10. Security. Analytics. Insight.10 • Objective: Find attackers in the network moving laterally • Defines data needed (netflow, sflow, …) • maybe restrict to a network segment • Audience: security analyst, risk team, … • Informs how to visualize / present data For Example - Lateral Movement Recon Weaponize Deliver Exploit Install C2 Act
  • 11. Security. Analytics. Insight.11 • Show  comparisons, contrasts, differences • Show  causality, mechanism, explanation, systematic structure. • Show  multivariate data; that is, show more than 1 or 2 variables. by Edward Tufte Principals of Analytic Design
  • 13. Security. Analytics. Insight. 42 is just a number and means nothing without context 13 Show Context
  • 14.
  • 15. Security. Analytics. Insight.15 Use Numbers To Highlight Most Important Parts of Data Numbers Summaries
  • 16. Security. Analytics. Insight.16 Additional information about objects, such as: • machine • roles • criticality • location • owner • … • user • roles • office location • … Add Context source destination machine and 
 user context machine role user role
  • 17. Security. Analytics. Insight.17 Traffic Flow Analysis With Context
  • 18. Security. Analytics. Insight.18 http://www.scifiinterfaces.com/ • Black background • Blue or green colors • Glow Aesthetics Matter
  • 21. Security. Analytics. Insight.21 • Audience, audience, audience! • Comprehensive Information (enough context) • Highlight important data • Use graphics when appropriate • Good choice of graphics and design • Aesthetically pleasing • Enough information to decide if action is necessary • No scrolling • Real-time vs. batch? (Refresh-rates) • Clear organization Dashboard Design Principles
  • 24. Security. Analytics. Insight.24 • Disappears too quickly • Analysts focus is on their own screens • SOC dashboard just distracts • Detailed information not legible • Put the detailed dashboards on the analysts screens! Dashboards For Discovery
  • 25. Security. Analytics. Insight.25 • Provide analyst with context • “What else is going on in the environment right now?” • Bring Into Focus • Turn something benign into something interesting • Disprove • Turn something interesting into something benign Use SOC Dashboard For Context Environment informs detection policies
  • 26. Security. Analytics. Insight.26 Show Comparisons Current Measure week prior
  • 27. Security. Analytics. Insight.27 • News feed summary (FS ISAC feeds, mailinglists, threat feeds) • Monitoring twitter or IRC for certain activity / keywords • Volumes or metrics (e.g., #firewall blocks, #IDS alerts, #failed transactions) • Top N metrics: • Top 10 suspicious users • Top 10 servers connecting outbound What To Put on Screens Provide context to individual security alerts http://raffy.ch/blog/2015/01/15/dashboards-in-the-security-opartions-center-soc/
  • 30. Security. Analytics. Insight.30 Information Visualization Mantra Overview Zoom / Filter Details on Demand Principle by Ben Shneiderman • summary / aggregation • data mining • signal detection (IDS, behavioral, etc.)
  • 31. Security. Analytics. Insight.31 • Access to data • Parsed data and data context • Data architecture for central data access and fast queries • Application of data mining (how?, what?, scalable, …) • Visualization tools that support • Complex visual types (||-coordinates, treemaps, 
 heat maps, link graphs) • Linked views • Data mining (clustering, …) • Collaboration, information sharing • Visual analytics workflow Visualization Challenges
  • 33. Security. Analytics. Insight.33 • One central location to store all cyber security data • “Data collected only once and third party software leveraging it” • Scalability and interoperability • More than deploying an off the shelf product from a vendor • Data use influences both data formats and technologies to store the data • search, analytics, relationships, and distributed processing • correlation, and statistical summarization • What to do with Context? Enrich or join? • Hard problems: • Parsing: can you re-parse? Common naming scheme! • Data store capabilities (search, analytics, distributed processing, etc.) • Access to data: SQL (even in Hadoop context), how can products access the data? The Big Data Lake
  • 34. Security. Analytics. Insight.34 Federated Data Access SIEM dispatcher SIEM 
 connector SIEM console Prod A AD / LDAP HR … IDS FW Prod B DBs Data Lake Caveats: • Dispatcher? • Standard access to dispatcher /
 products enabled • Data lake technology? SNMP
  • 35. Security. Analytics. Insight.35 Multiple Data Stores raw logs key-value structured real-time
 processing (un)-structured data context SQL s t o r a g e stats index queue distributed
 processing a c c e s s graph Caveat: • Need multiple types of 
 data stores
  • 36. Security. Analytics. Insight.36 Technologies (Example) raw logs key-value (Cassandra) columnar (parquet) real-time
 processing (Spark) (un)-structured data context SQL (Impala, SparkSQL) H D F S aggregates index (ES) queue (Kafka) distributed
 processing (Spark) a c c e s s graph (GraphX) Caveat: • No out of the box solution available
  • 37. Security. Analytics. Insight.37 SIEM Integration - Log Management First SIEM columnar or search engine
 or log management processing SIEM 
 connector raw logs SIEM console SQL or search
 interface processing filtering H D F S e.g., PIG parsing
  • 38. Security. Analytics. Insight.38 Simple SIEM Integration raw, csv, json flume log data SQL (Impala, with SerDe) H D F S SIEM 
 connector SIEM Requirement: • SIEM connector to forward text- based data to Flume. SQL interface Tableau, etc. SIEM console
  • 39. Security. Analytics. Insight.39 SIEM Integration - Advanced SIEM columnar (parquet) processing syslog data SQL (Impala, SparkSQL) H D F S index (ES) queue (Kafka) a c c e s s other data sources SIEM 
 connector raw logs SIEM console SQL and search 
 interface Tableau, Kibana, etc. requires parsing and formatting in a SIEM readable format (e.g., CEF)
  • 40. Security. Analytics. Insight.40 What I am Working On Data Stores Analytics Forensics Models Admin 10.9.79.109 --> 3.16.204.150 10.8.24.80 --> 192.168.148.193 10.8.50.85 --> 192.168.148.193 10.8.48.128 --> 192.168.148.193 10.9.79.6 --> 192.168.148.193 10.9.79.6 10.8.48.128 80 53 8.8.8.8 127.0.0.1 Anomalies Decomposition Data Seasonal Trend Anomaly Details “Hunt” ExplainVisual Search • Big data backend • Own visualization engine (Web-based) • Visualization workflows
  • 41. Security. Analytics. Insight.41 BlackHat Workshop Visual Analytics - Delivering Actionable Security Intelligence August 1-6 2015, Las Vegas, USA big data | analytics | visualization
  • 42. Security. Analytics. Insight.42 http://secviz.org List: secviz.org/mailinglist Twitter: @secviz Share, discuss, challenge, and learn about security visualization. Security Visualization Community