SlideShare a Scribd company logo

Cloud Log Analysis and Visualization

Cloud computing has changed the way businesses operate, the way businesses make money, and the way business have to protect their assets and information. More and more software applications are moving into the cloud. People are running their proxies in the cloud and soon you will be collecting your logs in the cloud. You shouldn't have to deal with log collection and log management. You should be able to focus your time on getting value out of the logs; to do log analysis and visualization. In this presentation we will explore how we can leverage the cloud to build security visualization tools. We will discuss some common visualization libraries and have a look at how they can be deployed to solve security problems. We will see how easy it is to quickly stand up such an application. To close the presentation, we will look at a number of security visualization examples that show how security data benefits from visual representations. For example, how can network traffic, firewall data, or IDS data be visualized effectively?

1 of 43
Download to read offline
Cloud-based Log Analysis and Visualization
                        RMLL 2010, Bordeaux, France
                                               mobile-166   My syslog




                          Raffael Marty - @zrlram
Tuesday, July 6, 2010
Raffael (Raffy) Marty
       • Founder @
       • Chief Security Strategist and Product Manager @ Splunk
       • Manager Solutions @ ArcSight
       • Intrusion Detection Research @ IBM Research
       • IT Security Consultant @ PriceWaterhouse Coopers
                           Applied Security Visualization
                               Publisher: Addison Wesley (August, 2008)
                                           ISBN: 0321510100




                        Logging as a Service                              2   (c) by Raffael Marty
Tuesday, July 6, 2010
Agenda
            •Introduction                            •Do it Yourself

            •Visualization                            •AfterGlow
                                                      •Google Visualization API
            •InfoViz Process
                                                     •Visualization Use-Cases
            •Visualization Tools
                                                     •Visualization Resources
            •The Cloud

            •Loggly

                        Logging as a Service     3                          (c) by Raffael Marty
Tuesday, July 6, 2010
Open Your Eyes




                        Logging as a Service   4         (c) by Raffael Marty
Tuesday, July 6, 2010
Security Is About Seeing




                        Logging as a Service   5   (c) by Raffael Marty
Tuesday, July 6, 2010
Goals
       - Learn how you can
          - use visualization to help solve security problems
          - leverage the cloud to build security visualization tools




                        Logging as a Service     6          (c) by Raffael Marty
Tuesday, July 6, 2010
Ad

Recommended

Cloud Application Logging for Forensics
Cloud Application Logging for ForensicsCloud Application Logging for Forensics
Cloud Application Logging for ForensicsRaffael Marty
 
Mining Your Logs - Gaining Insight Through Visualization
Mining Your Logs - Gaining Insight Through VisualizationMining Your Logs - Gaining Insight Through Visualization
Mining Your Logs - Gaining Insight Through VisualizationRaffael Marty
 
Making Hadoop Realtime by Dr. William Bain of Scaleout Software
Making Hadoop Realtime by Dr. William Bain of Scaleout SoftwareMaking Hadoop Realtime by Dr. William Bain of Scaleout Software
Making Hadoop Realtime by Dr. William Bain of Scaleout SoftwareData Con LA
 
Scalable Preservation Workflows
Scalable Preservation WorkflowsScalable Preservation Workflows
Scalable Preservation WorkflowsSCAPE Project
 
Data pipelines from zero
Data pipelines from zero Data pipelines from zero
Data pipelines from zero Lars Albertsson
 
RUNNING A PETASCALE DATA SYSTEM: GOOD, BAD, AND UGLY CHOICES by Alexey Kharlamov
RUNNING A PETASCALE DATA SYSTEM: GOOD, BAD, AND UGLY CHOICES by Alexey KharlamovRUNNING A PETASCALE DATA SYSTEM: GOOD, BAD, AND UGLY CHOICES by Alexey Kharlamov
RUNNING A PETASCALE DATA SYSTEM: GOOD, BAD, AND UGLY CHOICES by Alexey KharlamovBig Data Spain
 

More Related Content

What's hot

Fast and Scalable Python
Fast and Scalable PythonFast and Scalable Python
Fast and Scalable PythonTravis Oliphant
 
How to Use Innovative Data Handling and Processing Techniques to Drive Alpha ...
How to Use Innovative Data Handling and Processing Techniques to Drive Alpha ...How to Use Innovative Data Handling and Processing Techniques to Drive Alpha ...
How to Use Innovative Data Handling and Processing Techniques to Drive Alpha ...DataWorks Summit
 
Using BigBench to compare Hive and Spark (Long version)
Using BigBench to compare Hive and Spark (Long version)Using BigBench to compare Hive and Spark (Long version)
Using BigBench to compare Hive and Spark (Long version)Nicolas Poggi
 
Drag and Drop Open Source GeoTools ETL with Apache NiFi
Drag and Drop Open Source GeoTools ETL with Apache NiFiDrag and Drop Open Source GeoTools ETL with Apache NiFi
Drag and Drop Open Source GeoTools ETL with Apache NiFi"Constantin \"Cristi\"" Stanca
 
The convergence of reporting and interactive BI on Hadoop
The convergence of reporting and interactive BI on HadoopThe convergence of reporting and interactive BI on Hadoop
The convergence of reporting and interactive BI on HadoopDataWorks Summit
 
Lessons learned processing 70 billion data points a day using the hybrid cloud
Lessons learned processing 70 billion data points a day using the hybrid cloudLessons learned processing 70 billion data points a day using the hybrid cloud
Lessons learned processing 70 billion data points a day using the hybrid cloudDataWorks Summit
 
Db tech show - hivemall
Db tech show - hivemallDb tech show - hivemall
Db tech show - hivemallMakoto Yui
 
Introduction to the Hadoop EcoSystem
Introduction to the Hadoop EcoSystemIntroduction to the Hadoop EcoSystem
Introduction to the Hadoop EcoSystemShivaji Dutta
 
Bridging the gap: achieving fast data synchronization from SAP HANA by levera...
Bridging the gap: achieving fast data synchronization from SAP HANA by levera...Bridging the gap: achieving fast data synchronization from SAP HANA by levera...
Bridging the gap: achieving fast data synchronization from SAP HANA by levera...DataWorks Summit
 
Teradata Partners Conference Oct 2014 Big Data Anti-Patterns
Teradata Partners Conference Oct 2014   Big Data Anti-PatternsTeradata Partners Conference Oct 2014   Big Data Anti-Patterns
Teradata Partners Conference Oct 2014 Big Data Anti-PatternsDouglas Moore
 
Bringing an AI Ecosystem to the Domain Expert and Enterprise AI Developer wit...
Bringing an AI Ecosystem to the Domain Expert and Enterprise AI Developer wit...Bringing an AI Ecosystem to the Domain Expert and Enterprise AI Developer wit...
Bringing an AI Ecosystem to the Domain Expert and Enterprise AI Developer wit...Databricks
 
Apache Eagle: eBay构建开源分布式实时预警引擎实践
Apache Eagle: eBay构建开源分布式实时预警引擎实践Apache Eagle: eBay构建开源分布式实时预警引擎实践
Apache Eagle: eBay构建开源分布式实时预警引擎实践Hao Chen
 

What's hot (16)

Fast and Scalable Python
Fast and Scalable PythonFast and Scalable Python
Fast and Scalable Python
 
How to Use Innovative Data Handling and Processing Techniques to Drive Alpha ...
How to Use Innovative Data Handling and Processing Techniques to Drive Alpha ...How to Use Innovative Data Handling and Processing Techniques to Drive Alpha ...
How to Use Innovative Data Handling and Processing Techniques to Drive Alpha ...
 
Using BigBench to compare Hive and Spark (Long version)
Using BigBench to compare Hive and Spark (Long version)Using BigBench to compare Hive and Spark (Long version)
Using BigBench to compare Hive and Spark (Long version)
 
How do you decide where your customer was?
How do you decide where your customer was?How do you decide where your customer was?
How do you decide where your customer was?
 
Benchmarking
BenchmarkingBenchmarking
Benchmarking
 
Time-oriented event search. A new level of scale
Time-oriented event search. A new level of scale Time-oriented event search. A new level of scale
Time-oriented event search. A new level of scale
 
London level39
London level39London level39
London level39
 
Drag and Drop Open Source GeoTools ETL with Apache NiFi
Drag and Drop Open Source GeoTools ETL with Apache NiFiDrag and Drop Open Source GeoTools ETL with Apache NiFi
Drag and Drop Open Source GeoTools ETL with Apache NiFi
 
The convergence of reporting and interactive BI on Hadoop
The convergence of reporting and interactive BI on HadoopThe convergence of reporting and interactive BI on Hadoop
The convergence of reporting and interactive BI on Hadoop
 
Lessons learned processing 70 billion data points a day using the hybrid cloud
Lessons learned processing 70 billion data points a day using the hybrid cloudLessons learned processing 70 billion data points a day using the hybrid cloud
Lessons learned processing 70 billion data points a day using the hybrid cloud
 
Db tech show - hivemall
Db tech show - hivemallDb tech show - hivemall
Db tech show - hivemall
 
Introduction to the Hadoop EcoSystem
Introduction to the Hadoop EcoSystemIntroduction to the Hadoop EcoSystem
Introduction to the Hadoop EcoSystem
 
Bridging the gap: achieving fast data synchronization from SAP HANA by levera...
Bridging the gap: achieving fast data synchronization from SAP HANA by levera...Bridging the gap: achieving fast data synchronization from SAP HANA by levera...
Bridging the gap: achieving fast data synchronization from SAP HANA by levera...
 
Teradata Partners Conference Oct 2014 Big Data Anti-Patterns
Teradata Partners Conference Oct 2014   Big Data Anti-PatternsTeradata Partners Conference Oct 2014   Big Data Anti-Patterns
Teradata Partners Conference Oct 2014 Big Data Anti-Patterns
 
Bringing an AI Ecosystem to the Domain Expert and Enterprise AI Developer wit...
Bringing an AI Ecosystem to the Domain Expert and Enterprise AI Developer wit...Bringing an AI Ecosystem to the Domain Expert and Enterprise AI Developer wit...
Bringing an AI Ecosystem to the Domain Expert and Enterprise AI Developer wit...
 
Apache Eagle: eBay构建开源分布式实时预警引擎实践
Apache Eagle: eBay构建开源分布式实时预警引擎实践Apache Eagle: eBay构建开源分布式实时预警引擎实践
Apache Eagle: eBay构建开源分布式实时预警引擎实践
 

Viewers also liked

What Is Log Analyis
What Is Log AnalyisWhat Is Log Analyis
What Is Log AnalyisJim Jansen
 
Warehouse based Intelligent Banking Transaction Analysis System
Warehouse based Intelligent Banking Transaction Analysis SystemWarehouse based Intelligent Banking Transaction Analysis System
Warehouse based Intelligent Banking Transaction Analysis SystemJivan Nepali
 
Crystal Ball Event Prediction and Log Analysis with Hadoop MapReduce and Spark
Crystal Ball Event Prediction and Log Analysis with Hadoop MapReduce and SparkCrystal Ball Event Prediction and Log Analysis with Hadoop MapReduce and Spark
Crystal Ball Event Prediction and Log Analysis with Hadoop MapReduce and SparkJivan Nepali
 
0610 w13 ms_61
0610 w13 ms_610610 w13 ms_61
0610 w13 ms_61King Ali
 
A Basic Guide to Server Log Analysis
A Basic Guide to Server Log AnalysisA Basic Guide to Server Log Analysis
A Basic Guide to Server Log AnalysisAndrew Halliday
 
Debugging Skynet: A Machine Learning Approach to Log Analysis - Ianir Ideses,...
Debugging Skynet: A Machine Learning Approach to Log Analysis - Ianir Ideses,...Debugging Skynet: A Machine Learning Approach to Log Analysis - Ianir Ideses,...
Debugging Skynet: A Machine Learning Approach to Log Analysis - Ianir Ideses,...DevOpsDays Tel Aviv
 
Building Product from ground up using Open Source Technologies
Building Product from ground up using Open Source TechnologiesBuilding Product from ground up using Open Source Technologies
Building Product from ground up using Open Source TechnologiesAmit Goel
 
Experiences in ELK with D3.js for Large Log Analysis and Visualization
Experiences in ELK with D3.js  for Large Log Analysis  and VisualizationExperiences in ELK with D3.js  for Large Log Analysis  and Visualization
Experiences in ELK with D3.js for Large Log Analysis and VisualizationSurasak Sanguanpong
 
Log analysis with Hadoop in livedoor 2013
Log analysis with Hadoop in livedoor 2013Log analysis with Hadoop in livedoor 2013
Log analysis with Hadoop in livedoor 2013SATOSHI TAGOMORI
 
Security Insights at Scale
Security Insights at ScaleSecurity Insights at Scale
Security Insights at ScaleRaffael Marty
 
Modern log yönetimi sistemleri ve trafik analizi
Modern log yönetimi sistemleri ve trafik analiziModern log yönetimi sistemleri ve trafik analizi
Modern log yönetimi sistemleri ve trafik analiziErtugrul Akbas
 
Log Korelasyon/SIEM Kural Örnekleri ve Korelasyon Motoru Performans Verileri
Log Korelasyon/SIEM Kural Örnekleri ve Korelasyon Motoru Performans VerileriLog Korelasyon/SIEM Kural Örnekleri ve Korelasyon Motoru Performans Verileri
Log Korelasyon/SIEM Kural Örnekleri ve Korelasyon Motoru Performans VerileriErtugrul Akbas
 
Log Yonetimi ve SIEM Kontrol Listesi
Log Yonetimi ve SIEM Kontrol Listesi Log Yonetimi ve SIEM Kontrol Listesi
Log Yonetimi ve SIEM Kontrol Listesi Ertugrul Akbas
 
LWV MV Info Brochure 2016 Web-1
LWV MV Info Brochure 2016 Web-1LWV MV Info Brochure 2016 Web-1
LWV MV Info Brochure 2016 Web-1Sarah Robinson
 
عربی کی چینی طور میں کیلی گرافی
عربی کی چینی طور میں کیلی گرافیعربی کی چینی طور میں کیلی گرافی
عربی کی چینی طور میں کیلی گرافیmaqsood hasni
 
Google Analytics and Webmaster tool
Google Analytics and Webmaster toolGoogle Analytics and Webmaster tool
Google Analytics and Webmaster toolRUBEN LICERA
 
New Technologies Close the Recruitment Gap
New Technologies Close the Recruitment GapNew Technologies Close the Recruitment Gap
New Technologies Close the Recruitment GapJohn Reites
 
Oracle on Oracle (OoO)
Oracle on Oracle (OoO)Oracle on Oracle (OoO)
Oracle on Oracle (OoO)Oracle
 

Viewers also liked (20)

What Is Log Analyis
What Is Log AnalyisWhat Is Log Analyis
What Is Log Analyis
 
Warehouse based Intelligent Banking Transaction Analysis System
Warehouse based Intelligent Banking Transaction Analysis SystemWarehouse based Intelligent Banking Transaction Analysis System
Warehouse based Intelligent Banking Transaction Analysis System
 
Crystal Ball Event Prediction and Log Analysis with Hadoop MapReduce and Spark
Crystal Ball Event Prediction and Log Analysis with Hadoop MapReduce and SparkCrystal Ball Event Prediction and Log Analysis with Hadoop MapReduce and Spark
Crystal Ball Event Prediction and Log Analysis with Hadoop MapReduce and Spark
 
0610 w13 ms_61
0610 w13 ms_610610 w13 ms_61
0610 w13 ms_61
 
A Basic Guide to Server Log Analysis
A Basic Guide to Server Log AnalysisA Basic Guide to Server Log Analysis
A Basic Guide to Server Log Analysis
 
Debugging Skynet: A Machine Learning Approach to Log Analysis - Ianir Ideses,...
Debugging Skynet: A Machine Learning Approach to Log Analysis - Ianir Ideses,...Debugging Skynet: A Machine Learning Approach to Log Analysis - Ianir Ideses,...
Debugging Skynet: A Machine Learning Approach to Log Analysis - Ianir Ideses,...
 
Building Product from ground up using Open Source Technologies
Building Product from ground up using Open Source TechnologiesBuilding Product from ground up using Open Source Technologies
Building Product from ground up using Open Source Technologies
 
Experiences in ELK with D3.js for Large Log Analysis and Visualization
Experiences in ELK with D3.js  for Large Log Analysis  and VisualizationExperiences in ELK with D3.js  for Large Log Analysis  and Visualization
Experiences in ELK with D3.js for Large Log Analysis and Visualization
 
Log Data Mining
Log Data MiningLog Data Mining
Log Data Mining
 
Log analysis with Hadoop in livedoor 2013
Log analysis with Hadoop in livedoor 2013Log analysis with Hadoop in livedoor 2013
Log analysis with Hadoop in livedoor 2013
 
Security Insights at Scale
Security Insights at ScaleSecurity Insights at Scale
Security Insights at Scale
 
Modern log yönetimi sistemleri ve trafik analizi
Modern log yönetimi sistemleri ve trafik analiziModern log yönetimi sistemleri ve trafik analizi
Modern log yönetimi sistemleri ve trafik analizi
 
Log siem korelasyon
Log siem korelasyonLog siem korelasyon
Log siem korelasyon
 
Log Korelasyon/SIEM Kural Örnekleri ve Korelasyon Motoru Performans Verileri
Log Korelasyon/SIEM Kural Örnekleri ve Korelasyon Motoru Performans VerileriLog Korelasyon/SIEM Kural Örnekleri ve Korelasyon Motoru Performans Verileri
Log Korelasyon/SIEM Kural Örnekleri ve Korelasyon Motoru Performans Verileri
 
Log Yonetimi ve SIEM Kontrol Listesi
Log Yonetimi ve SIEM Kontrol Listesi Log Yonetimi ve SIEM Kontrol Listesi
Log Yonetimi ve SIEM Kontrol Listesi
 
LWV MV Info Brochure 2016 Web-1
LWV MV Info Brochure 2016 Web-1LWV MV Info Brochure 2016 Web-1
LWV MV Info Brochure 2016 Web-1
 
عربی کی چینی طور میں کیلی گرافی
عربی کی چینی طور میں کیلی گرافیعربی کی چینی طور میں کیلی گرافی
عربی کی چینی طور میں کیلی گرافی
 
Google Analytics and Webmaster tool
Google Analytics and Webmaster toolGoogle Analytics and Webmaster tool
Google Analytics and Webmaster tool
 
New Technologies Close the Recruitment Gap
New Technologies Close the Recruitment GapNew Technologies Close the Recruitment Gap
New Technologies Close the Recruitment Gap
 
Oracle on Oracle (OoO)
Oracle on Oracle (OoO)Oracle on Oracle (OoO)
Oracle on Oracle (OoO)
 

Similar to Cloud Log Analysis and Visualization

The architecture of data analytics PaaS on AWS
The architecture of data analytics PaaS on AWSThe architecture of data analytics PaaS on AWS
The architecture of data analytics PaaS on AWSTreasure Data, Inc.
 
Open source for customer analytics
Open source for customer analyticsOpen source for customer analytics
Open source for customer analyticsMatthias Funke
 
Graphite tattle
Graphite tattleGraphite tattle
Graphite tattleDraco2002
 
Open@Fao presentation at the EADI Open For Development Project, 2012
Open@Fao presentation at the EADI Open For Development Project, 2012 Open@Fao presentation at the EADI Open For Development Project, 2012
Open@Fao presentation at the EADI Open For Development Project, 2012 Stephen Katz
 
Billions of hits: Scaling Twitter (Web 2.0 Expo, SF)
Billions of hits: Scaling Twitter (Web 2.0 Expo, SF)Billions of hits: Scaling Twitter (Web 2.0 Expo, SF)
Billions of hits: Scaling Twitter (Web 2.0 Expo, SF)John Adams
 
GeoSDI: una piattaforma social di dati geografici basata sui principi di INSP...
GeoSDI: una piattaforma social di dati geografici basata sui principi di INSP...GeoSDI: una piattaforma social di dati geografici basata sui principi di INSP...
GeoSDI: una piattaforma social di dati geografici basata sui principi di INSP...Beniamino Murgante
 
Red Dirt Ruby Conference
Red Dirt Ruby ConferenceRed Dirt Ruby Conference
Red Dirt Ruby ConferenceJohn Woodell
 
ONLINE IMAGE PROCESSING WITH ORFEOTOOLBOX WPS
ONLINE IMAGE PROCESSING WITH ORFEOTOOLBOX WPSONLINE IMAGE PROCESSING WITH ORFEOTOOLBOX WPS
ONLINE IMAGE PROCESSING WITH ORFEOTOOLBOX WPSotb
 
RESTFul Services, Does it Matter Anymore?
RESTFul Services, Does it Matter Anymore?RESTFul Services, Does it Matter Anymore?
RESTFul Services, Does it Matter Anymore?Pat Cappelaere
 
EOSC-hub and OpenAIRE Advance webinar - introduction
EOSC-hub and OpenAIRE Advance webinar - introductionEOSC-hub and OpenAIRE Advance webinar - introduction
EOSC-hub and OpenAIRE Advance webinar - introductionOpenAIRE
 
AFCEA C4I Symposium: The 4th C in C4I Stands for Cloud:Factors Driving Adopti...
AFCEA C4I Symposium: The 4th C in C4I Stands for Cloud:Factors Driving Adopti...AFCEA C4I Symposium: The 4th C in C4I Stands for Cloud:Factors Driving Adopti...
AFCEA C4I Symposium: The 4th C in C4I Stands for Cloud:Factors Driving Adopti...Patrick Chanezon
 
Tools & Measurements
Tools & MeasurementsTools & Measurements
Tools & MeasurementsRIPE NCC
 
RIPE NCC Measurements Tools
RIPE NCC Measurements ToolsRIPE NCC Measurements Tools
RIPE NCC Measurements ToolsRIPE NCC
 
IDB-Cloud Providing Bioinformatics Services on Cloud
IDB-Cloud Providing Bioinformatics Services on CloudIDB-Cloud Providing Bioinformatics Services on Cloud
IDB-Cloud Providing Bioinformatics Services on Cloudstratuslab
 
Geosolutions gwf-2015-v01.04
Geosolutions gwf-2015-v01.04Geosolutions gwf-2015-v01.04
Geosolutions gwf-2015-v01.04GeoSolutions
 
RIPEstat Demo - s2e03
RIPEstat Demo - s2e03RIPEstat Demo - s2e03
RIPEstat Demo - s2e03RIPE NCC
 
Why and How to integrate Hadoop and NoSQL?
Why and How to integrate Hadoop and NoSQL?Why and How to integrate Hadoop and NoSQL?
Why and How to integrate Hadoop and NoSQL?Tugdual Grall
 
SpagoBI 5 Demo Day and Workshop : Technology Applications and Uses
SpagoBI 5 Demo Day and Workshop : Technology Applications and UsesSpagoBI 5 Demo Day and Workshop : Technology Applications and Uses
SpagoBI 5 Demo Day and Workshop : Technology Applications and UsesSpagoWorld
 

Similar to Cloud Log Analysis and Visualization (20)

The architecture of data analytics PaaS on AWS
The architecture of data analytics PaaS on AWSThe architecture of data analytics PaaS on AWS
The architecture of data analytics PaaS on AWS
 
Open source for customer analytics
Open source for customer analyticsOpen source for customer analytics
Open source for customer analytics
 
App Engine Meetup
App Engine MeetupApp Engine Meetup
App Engine Meetup
 
Graphite tattle
Graphite tattleGraphite tattle
Graphite tattle
 
Open@Fao presentation at the EADI Open For Development Project, 2012
Open@Fao presentation at the EADI Open For Development Project, 2012 Open@Fao presentation at the EADI Open For Development Project, 2012
Open@Fao presentation at the EADI Open For Development Project, 2012
 
Billions of hits: Scaling Twitter (Web 2.0 Expo, SF)
Billions of hits: Scaling Twitter (Web 2.0 Expo, SF)Billions of hits: Scaling Twitter (Web 2.0 Expo, SF)
Billions of hits: Scaling Twitter (Web 2.0 Expo, SF)
 
GeoSDI: una piattaforma social di dati geografici basata sui principi di INSP...
GeoSDI: una piattaforma social di dati geografici basata sui principi di INSP...GeoSDI: una piattaforma social di dati geografici basata sui principi di INSP...
GeoSDI: una piattaforma social di dati geografici basata sui principi di INSP...
 
Red Dirt Ruby Conference
Red Dirt Ruby ConferenceRed Dirt Ruby Conference
Red Dirt Ruby Conference
 
ONLINE IMAGE PROCESSING WITH ORFEOTOOLBOX WPS
ONLINE IMAGE PROCESSING WITH ORFEOTOOLBOX WPSONLINE IMAGE PROCESSING WITH ORFEOTOOLBOX WPS
ONLINE IMAGE PROCESSING WITH ORFEOTOOLBOX WPS
 
RESTFul Services, Does it Matter Anymore?
RESTFul Services, Does it Matter Anymore?RESTFul Services, Does it Matter Anymore?
RESTFul Services, Does it Matter Anymore?
 
EOSC-hub and OpenAIRE Advance webinar - introduction
EOSC-hub and OpenAIRE Advance webinar - introductionEOSC-hub and OpenAIRE Advance webinar - introduction
EOSC-hub and OpenAIRE Advance webinar - introduction
 
AFCEA C4I Symposium: The 4th C in C4I Stands for Cloud:Factors Driving Adopti...
AFCEA C4I Symposium: The 4th C in C4I Stands for Cloud:Factors Driving Adopti...AFCEA C4I Symposium: The 4th C in C4I Stands for Cloud:Factors Driving Adopti...
AFCEA C4I Symposium: The 4th C in C4I Stands for Cloud:Factors Driving Adopti...
 
Tools & Measurements
Tools & MeasurementsTools & Measurements
Tools & Measurements
 
RIPE NCC Measurements Tools
RIPE NCC Measurements ToolsRIPE NCC Measurements Tools
RIPE NCC Measurements Tools
 
IDB-Cloud Providing Bioinformatics Services on Cloud
IDB-Cloud Providing Bioinformatics Services on CloudIDB-Cloud Providing Bioinformatics Services on Cloud
IDB-Cloud Providing Bioinformatics Services on Cloud
 
Geosolutions gwf-2015-v01.04
Geosolutions gwf-2015-v01.04Geosolutions gwf-2015-v01.04
Geosolutions gwf-2015-v01.04
 
RIPEstat Demo - s2e03
RIPEstat Demo - s2e03RIPEstat Demo - s2e03
RIPEstat Demo - s2e03
 
Why and How to integrate Hadoop and NoSQL?
Why and How to integrate Hadoop and NoSQL?Why and How to integrate Hadoop and NoSQL?
Why and How to integrate Hadoop and NoSQL?
 
SpagoBI 5 Demo Day and Workshop : Technology Applications and Uses
SpagoBI 5 Demo Day and Workshop : Technology Applications and UsesSpagoBI 5 Demo Day and Workshop : Technology Applications and Uses
SpagoBI 5 Demo Day and Workshop : Technology Applications and Uses
 
20100608sigmod
20100608sigmod20100608sigmod
20100608sigmod
 

More from Raffael Marty

Exploring the Defender's Advantage
Exploring the Defender's AdvantageExploring the Defender's Advantage
Exploring the Defender's AdvantageRaffael Marty
 
Extended Detection and Response (XDR) An Overhyped Product Category With Ulti...
Extended Detection and Response (XDR)An Overhyped Product Category With Ulti...Extended Detection and Response (XDR)An Overhyped Product Category With Ulti...
Extended Detection and Response (XDR) An Overhyped Product Category With Ulti...Raffael Marty
 
How To Drive Value with Security Data
How To Drive Value with Security DataHow To Drive Value with Security Data
How To Drive Value with Security DataRaffael Marty
 
Cyber Security Beyond 2020 – Will We Learn From Our Mistakes?
Cyber Security Beyond 2020 – Will We Learn From Our Mistakes?Cyber Security Beyond 2020 – Will We Learn From Our Mistakes?
Cyber Security Beyond 2020 – Will We Learn From Our Mistakes?Raffael Marty
 
Artificial Intelligence – Time Bomb or The Promised Land?
Artificial Intelligence – Time Bomb or The Promised Land?Artificial Intelligence – Time Bomb or The Promised Land?
Artificial Intelligence – Time Bomb or The Promised Land?Raffael Marty
 
Understanding the "Intelligence" in AI
Understanding the "Intelligence" in AIUnderstanding the "Intelligence" in AI
Understanding the "Intelligence" in AIRaffael Marty
 
AI & ML in Cyber Security - Why Algorithms are Dangerous
AI & ML in Cyber Security - Why Algorithms are DangerousAI & ML in Cyber Security - Why Algorithms are Dangerous
AI & ML in Cyber Security - Why Algorithms are DangerousRaffael Marty
 
AI & ML in Cyber Security - Why Algorithms Are Dangerous
AI & ML in Cyber Security - Why Algorithms Are DangerousAI & ML in Cyber Security - Why Algorithms Are Dangerous
AI & ML in Cyber Security - Why Algorithms Are DangerousRaffael Marty
 
Delivering Security Insights with Data Analytics and Visualization
Delivering Security Insights with Data Analytics and VisualizationDelivering Security Insights with Data Analytics and Visualization
Delivering Security Insights with Data Analytics and VisualizationRaffael Marty
 
AI & ML in Cyber Security - Welcome Back to 1999 - Security Hasn't Changed
AI & ML in Cyber Security - Welcome Back to 1999 - Security Hasn't ChangedAI & ML in Cyber Security - Welcome Back to 1999 - Security Hasn't Changed
AI & ML in Cyber Security - Welcome Back to 1999 - Security Hasn't ChangedRaffael Marty
 
Creating Your Own Threat Intel Through Hunting & Visualization
Creating Your Own Threat Intel Through Hunting & VisualizationCreating Your Own Threat Intel Through Hunting & Visualization
Creating Your Own Threat Intel Through Hunting & VisualizationRaffael Marty
 
Creating Your Own Threat Intel Through Hunting & Visualization
Creating Your Own Threat Intel Through Hunting & VisualizationCreating Your Own Threat Intel Through Hunting & Visualization
Creating Your Own Threat Intel Through Hunting & VisualizationRaffael Marty
 
Visualization in the Age of Big Data
Visualization in the Age of Big DataVisualization in the Age of Big Data
Visualization in the Age of Big DataRaffael Marty
 
Big Data Visualization
Big Data VisualizationBig Data Visualization
Big Data VisualizationRaffael Marty
 
The Heatmap
 - Why is Security Visualization so Hard?
The Heatmap
 - Why is Security Visualization so Hard?The Heatmap
 - Why is Security Visualization so Hard?
The Heatmap
 - Why is Security Visualization so Hard?Raffael Marty
 
Workshop: Big Data Visualization for Security
Workshop: Big Data Visualization for SecurityWorkshop: Big Data Visualization for Security
Workshop: Big Data Visualization for SecurityRaffael Marty
 
Visualization for Security
Visualization for SecurityVisualization for Security
Visualization for SecurityRaffael Marty
 
The Heatmap
 - Why is Security Visualization so Hard?
The Heatmap
 - Why is Security Visualization so Hard?The Heatmap
 - Why is Security Visualization so Hard?
The Heatmap
 - Why is Security Visualization so Hard?Raffael Marty
 
DAVIX - Data Analysis and Visualization Linux
DAVIX - Data Analysis and Visualization LinuxDAVIX - Data Analysis and Visualization Linux
DAVIX - Data Analysis and Visualization LinuxRaffael Marty
 

More from Raffael Marty (20)

Exploring the Defender's Advantage
Exploring the Defender's AdvantageExploring the Defender's Advantage
Exploring the Defender's Advantage
 
Extended Detection and Response (XDR) An Overhyped Product Category With Ulti...
Extended Detection and Response (XDR)An Overhyped Product Category With Ulti...Extended Detection and Response (XDR)An Overhyped Product Category With Ulti...
Extended Detection and Response (XDR) An Overhyped Product Category With Ulti...
 
How To Drive Value with Security Data
How To Drive Value with Security DataHow To Drive Value with Security Data
How To Drive Value with Security Data
 
Cyber Security Beyond 2020 – Will We Learn From Our Mistakes?
Cyber Security Beyond 2020 – Will We Learn From Our Mistakes?Cyber Security Beyond 2020 – Will We Learn From Our Mistakes?
Cyber Security Beyond 2020 – Will We Learn From Our Mistakes?
 
Artificial Intelligence – Time Bomb or The Promised Land?
Artificial Intelligence – Time Bomb or The Promised Land?Artificial Intelligence – Time Bomb or The Promised Land?
Artificial Intelligence – Time Bomb or The Promised Land?
 
Understanding the "Intelligence" in AI
Understanding the "Intelligence" in AIUnderstanding the "Intelligence" in AI
Understanding the "Intelligence" in AI
 
Security Chat 5.0
Security Chat 5.0Security Chat 5.0
Security Chat 5.0
 
AI & ML in Cyber Security - Why Algorithms are Dangerous
AI & ML in Cyber Security - Why Algorithms are DangerousAI & ML in Cyber Security - Why Algorithms are Dangerous
AI & ML in Cyber Security - Why Algorithms are Dangerous
 
AI & ML in Cyber Security - Why Algorithms Are Dangerous
AI & ML in Cyber Security - Why Algorithms Are DangerousAI & ML in Cyber Security - Why Algorithms Are Dangerous
AI & ML in Cyber Security - Why Algorithms Are Dangerous
 
Delivering Security Insights with Data Analytics and Visualization
Delivering Security Insights with Data Analytics and VisualizationDelivering Security Insights with Data Analytics and Visualization
Delivering Security Insights with Data Analytics and Visualization
 
AI & ML in Cyber Security - Welcome Back to 1999 - Security Hasn't Changed
AI & ML in Cyber Security - Welcome Back to 1999 - Security Hasn't ChangedAI & ML in Cyber Security - Welcome Back to 1999 - Security Hasn't Changed
AI & ML in Cyber Security - Welcome Back to 1999 - Security Hasn't Changed
 
Creating Your Own Threat Intel Through Hunting & Visualization
Creating Your Own Threat Intel Through Hunting & VisualizationCreating Your Own Threat Intel Through Hunting & Visualization
Creating Your Own Threat Intel Through Hunting & Visualization
 
Creating Your Own Threat Intel Through Hunting & Visualization
Creating Your Own Threat Intel Through Hunting & VisualizationCreating Your Own Threat Intel Through Hunting & Visualization
Creating Your Own Threat Intel Through Hunting & Visualization
 
Visualization in the Age of Big Data
Visualization in the Age of Big DataVisualization in the Age of Big Data
Visualization in the Age of Big Data
 
Big Data Visualization
Big Data VisualizationBig Data Visualization
Big Data Visualization
 
The Heatmap
 - Why is Security Visualization so Hard?
The Heatmap
 - Why is Security Visualization so Hard?The Heatmap
 - Why is Security Visualization so Hard?
The Heatmap
 - Why is Security Visualization so Hard?
 
Workshop: Big Data Visualization for Security
Workshop: Big Data Visualization for SecurityWorkshop: Big Data Visualization for Security
Workshop: Big Data Visualization for Security
 
Visualization for Security
Visualization for SecurityVisualization for Security
Visualization for Security
 
The Heatmap
 - Why is Security Visualization so Hard?
The Heatmap
 - Why is Security Visualization so Hard?The Heatmap
 - Why is Security Visualization so Hard?
The Heatmap
 - Why is Security Visualization so Hard?
 
DAVIX - Data Analysis and Visualization Linux
DAVIX - Data Analysis and Visualization LinuxDAVIX - Data Analysis and Visualization Linux
DAVIX - Data Analysis and Visualization Linux
 

Recently uploaded

IT Nation Evolve event 2024 - Quarter 1
IT Nation Evolve event 2024  - Quarter 1IT Nation Evolve event 2024  - Quarter 1
IT Nation Evolve event 2024 - Quarter 1Inbay UK
 
Artificial-Intelligence-in-Marketing-Data.pdf
Artificial-Intelligence-in-Marketing-Data.pdfArtificial-Intelligence-in-Marketing-Data.pdf
Artificial-Intelligence-in-Marketing-Data.pdfIsidro Navarro
 
2024 February Patch Tuesday
2024 February Patch Tuesday2024 February Patch Tuesday
2024 February Patch TuesdayIvanti
 
"Platform Engineering with Development Containers", Igor Fesenko
"Platform Engineering with Development Containers", Igor Fesenko"Platform Engineering with Development Containers", Igor Fesenko
"Platform Engineering with Development Containers", Igor FesenkoFwdays
 
Building Products That Think- Bhaskaran Srinivasan & Ashish Gupta
Building Products That Think- Bhaskaran Srinivasan & Ashish GuptaBuilding Products That Think- Bhaskaran Srinivasan & Ashish Gupta
Building Products That Think- Bhaskaran Srinivasan & Ashish GuptaISPMAIndia
 
Confoo 2024 Gettings started with OpenAI and data science
Confoo 2024 Gettings started with OpenAI and data scienceConfoo 2024 Gettings started with OpenAI and data science
Confoo 2024 Gettings started with OpenAI and data scienceSusan Ibach
 
My self introduction to know others abut me
My self  introduction to know others abut meMy self  introduction to know others abut me
My self introduction to know others abut meManoj Prabakar B
 
Automate Your Master Data Processes for Shared Service Center Excellence
Automate Your Master Data Processes for Shared Service Center ExcellenceAutomate Your Master Data Processes for Shared Service Center Excellence
Automate Your Master Data Processes for Shared Service Center ExcellencePrecisely
 
How to write an effective Cyber Incident Response Plan
How to write an effective Cyber Incident Response PlanHow to write an effective Cyber Incident Response Plan
How to write an effective Cyber Incident Response PlanDatabarracks
 
Importance of magazines in education ppt
Importance of magazines in education pptImportance of magazines in education ppt
Importance of magazines in education pptsafnarafeek2002
 
Enhancing Productivity and Insight A Tour of JDK Tools Progress Beyond Java 17
Enhancing Productivity and Insight  A Tour of JDK Tools Progress Beyond Java 17Enhancing Productivity and Insight  A Tour of JDK Tools Progress Beyond Java 17
Enhancing Productivity and Insight A Tour of JDK Tools Progress Beyond Java 17Ana-Maria Mihalceanu
 
AI Act & Standardization: UNINFO involvement
AI Act & Standardization: UNINFO involvementAI Act & Standardization: UNINFO involvement
AI Act & Standardization: UNINFO involvementMimmo Squillace
 
LF Energy Webinar: Introduction to TROLIE
LF Energy Webinar: Introduction to TROLIELF Energy Webinar: Introduction to TROLIE
LF Energy Webinar: Introduction to TROLIEDanBrown980551
 
Automation Ops Series: Session 1 - Introduction and setup DevOps for UiPath p...
Automation Ops Series: Session 1 - Introduction and setup DevOps for UiPath p...Automation Ops Series: Session 1 - Introduction and setup DevOps for UiPath p...
Automation Ops Series: Session 1 - Introduction and setup DevOps for UiPath p...DianaGray10
 
Unlocking the Cloud's True Potential: Why Multitenancy Is The Key?
Unlocking the Cloud's True Potential: Why Multitenancy Is The Key?Unlocking the Cloud's True Potential: Why Multitenancy Is The Key?
Unlocking the Cloud's True Potential: Why Multitenancy Is The Key?GleecusTechlabs1
 
Bit N Build Poland
Bit N Build PolandBit N Build Poland
Bit N Build PolandGDSC PJATK
 
Power of 2024 - WITforce Odyssey.pptx.pdf
Power of 2024 - WITforce Odyssey.pptx.pdfPower of 2024 - WITforce Odyssey.pptx.pdf
Power of 2024 - WITforce Odyssey.pptx.pdfkatalinjordans1
 
Dynamical systems simulation in Python for science and engineering
Dynamical systems simulation in Python for science and engineeringDynamical systems simulation in Python for science and engineering
Dynamical systems simulation in Python for science and engineeringMassimo Talia
 
"The Transformative Power of AI and Open Challenges" by Dr. Manish Gupta, Google
"The Transformative Power of AI and Open Challenges" by Dr. Manish Gupta, Google"The Transformative Power of AI and Open Challenges" by Dr. Manish Gupta, Google
"The Transformative Power of AI and Open Challenges" by Dr. Manish Gupta, GoogleISPMAIndia
 

Recently uploaded (20)

IT Nation Evolve event 2024 - Quarter 1
IT Nation Evolve event 2024  - Quarter 1IT Nation Evolve event 2024  - Quarter 1
IT Nation Evolve event 2024 - Quarter 1
 
Artificial-Intelligence-in-Marketing-Data.pdf
Artificial-Intelligence-in-Marketing-Data.pdfArtificial-Intelligence-in-Marketing-Data.pdf
Artificial-Intelligence-in-Marketing-Data.pdf
 
2024 February Patch Tuesday
2024 February Patch Tuesday2024 February Patch Tuesday
2024 February Patch Tuesday
 
"Platform Engineering with Development Containers", Igor Fesenko
"Platform Engineering with Development Containers", Igor Fesenko"Platform Engineering with Development Containers", Igor Fesenko
"Platform Engineering with Development Containers", Igor Fesenko
 
Building Products That Think- Bhaskaran Srinivasan & Ashish Gupta
Building Products That Think- Bhaskaran Srinivasan & Ashish GuptaBuilding Products That Think- Bhaskaran Srinivasan & Ashish Gupta
Building Products That Think- Bhaskaran Srinivasan & Ashish Gupta
 
Confoo 2024 Gettings started with OpenAI and data science
Confoo 2024 Gettings started with OpenAI and data scienceConfoo 2024 Gettings started with OpenAI and data science
Confoo 2024 Gettings started with OpenAI and data science
 
My self introduction to know others abut me
My self  introduction to know others abut meMy self  introduction to know others abut me
My self introduction to know others abut me
 
Automate Your Master Data Processes for Shared Service Center Excellence
Automate Your Master Data Processes for Shared Service Center ExcellenceAutomate Your Master Data Processes for Shared Service Center Excellence
Automate Your Master Data Processes for Shared Service Center Excellence
 
How to write an effective Cyber Incident Response Plan
How to write an effective Cyber Incident Response PlanHow to write an effective Cyber Incident Response Plan
How to write an effective Cyber Incident Response Plan
 
Importance of magazines in education ppt
Importance of magazines in education pptImportance of magazines in education ppt
Importance of magazines in education ppt
 
Enhancing Productivity and Insight A Tour of JDK Tools Progress Beyond Java 17
Enhancing Productivity and Insight  A Tour of JDK Tools Progress Beyond Java 17Enhancing Productivity and Insight  A Tour of JDK Tools Progress Beyond Java 17
Enhancing Productivity and Insight A Tour of JDK Tools Progress Beyond Java 17
 
AI Act & Standardization: UNINFO involvement
AI Act & Standardization: UNINFO involvementAI Act & Standardization: UNINFO involvement
AI Act & Standardization: UNINFO involvement
 
LF Energy Webinar: Introduction to TROLIE
LF Energy Webinar: Introduction to TROLIELF Energy Webinar: Introduction to TROLIE
LF Energy Webinar: Introduction to TROLIE
 
Automation Ops Series: Session 1 - Introduction and setup DevOps for UiPath p...
Automation Ops Series: Session 1 - Introduction and setup DevOps for UiPath p...Automation Ops Series: Session 1 - Introduction and setup DevOps for UiPath p...
Automation Ops Series: Session 1 - Introduction and setup DevOps for UiPath p...
 
Unlocking the Cloud's True Potential: Why Multitenancy Is The Key?
Unlocking the Cloud's True Potential: Why Multitenancy Is The Key?Unlocking the Cloud's True Potential: Why Multitenancy Is The Key?
Unlocking the Cloud's True Potential: Why Multitenancy Is The Key?
 
Bit N Build Poland
Bit N Build PolandBit N Build Poland
Bit N Build Poland
 
Power of 2024 - WITforce Odyssey.pptx.pdf
Power of 2024 - WITforce Odyssey.pptx.pdfPower of 2024 - WITforce Odyssey.pptx.pdf
Power of 2024 - WITforce Odyssey.pptx.pdf
 
Dynamical systems simulation in Python for science and engineering
Dynamical systems simulation in Python for science and engineeringDynamical systems simulation in Python for science and engineering
Dynamical systems simulation in Python for science and engineering
 
"The Transformative Power of AI and Open Challenges" by Dr. Manish Gupta, Google
"The Transformative Power of AI and Open Challenges" by Dr. Manish Gupta, Google"The Transformative Power of AI and Open Challenges" by Dr. Manish Gupta, Google
"The Transformative Power of AI and Open Challenges" by Dr. Manish Gupta, Google
 
5 Tech Trend to Notice in ESG Landscape- 47Billion
5 Tech Trend to Notice in ESG Landscape- 47Billion5 Tech Trend to Notice in ESG Landscape- 47Billion
5 Tech Trend to Notice in ESG Landscape- 47Billion
 

Cloud Log Analysis and Visualization

  • 1. Cloud-based Log Analysis and Visualization RMLL 2010, Bordeaux, France mobile-166 My syslog Raffael Marty - @zrlram Tuesday, July 6, 2010
  • 2. Raffael (Raffy) Marty • Founder @ • Chief Security Strategist and Product Manager @ Splunk • Manager Solutions @ ArcSight • Intrusion Detection Research @ IBM Research • IT Security Consultant @ PriceWaterhouse Coopers Applied Security Visualization Publisher: Addison Wesley (August, 2008) ISBN: 0321510100 Logging as a Service 2 (c) by Raffael Marty Tuesday, July 6, 2010
  • 3. Agenda •Introduction •Do it Yourself •Visualization •AfterGlow •Google Visualization API •InfoViz Process •Visualization Use-Cases •Visualization Tools •Visualization Resources •The Cloud •Loggly Logging as a Service 3 (c) by Raffael Marty Tuesday, July 6, 2010
  • 4. Open Your Eyes Logging as a Service 4 (c) by Raffael Marty Tuesday, July 6, 2010
  • 5. Security Is About Seeing Logging as a Service 5 (c) by Raffael Marty Tuesday, July 6, 2010
  • 6. Goals - Learn how you can - use visualization to help solve security problems - leverage the cloud to build security visualization tools Logging as a Service 6 (c) by Raffael Marty Tuesday, July 6, 2010
  • 7. Information Visualization? A picture is worth a thousand log records. Inspire Explore and Discover Answer a Pose a New Increase Communicate Support Question Question Efficiency Information Decisions Logging as a Service 7 (c) by Raffael Marty Tuesday, July 6, 2010
  • 8. Visualization and The Cloud 8 Tuesday, July 6, 2010
  • 9. InfoViz Process Collect Process Visualize •large-scale data collection •Your parsers •Visualization Tools •and processing •Standard formats •and Libraries Logging as a Service 9 (c) by Raffael Marty Tuesday, July 6, 2010
  • 10. Collect 10 Tuesday, July 6, 2010
  • 11. Log Management • Log Collection and Centralization • Log Storage • Log Filtering • Log Aggregation • Log Search and Extraction • Log Retention and Archiving Logging as a Service 11 (c) by Raffael Marty Tuesday, July 6, 2010
  • 12. Process 12 Tuesday, July 6, 2010
  • 13. Standard Formats • Multiple formats Oct 13 20:00:43.874401 rule 193/0(match): block in on xl0: 212.251.89.126.3859 >: S 1818630320:1818630320(0) win 65535 <mss 1460,nop,nop,sackOK> (DF) Oct 13 20:00:43 fwbox local4:warn|warning fw07 %PIX-4-106023: Deny tcp src internet: 212.251.89.126/3859 dst 212.254.110.98/135 by access-group "internet_access_in" Oct 13 20:00:43 fwbox kernel: DROPPED IN=eth0 OUT= MAC=ff:ff:ff:ff:ff:ff:00:0f:cc: 81:40:94:08:00 SRC=212.251.89.126 DST=212.254.110.98 LEN=576 TOS=0x00 PREC=0x00 TTL=255 ID=8624 PROTO=TCP SPT=3859 DPT=135 LEN=556 • Log Standards ‣ CEE (cee.mitre.org) ‣ SDEE ‣ WELF ‣ IDMEF ‣ CBE ‣ XDAS Logging as a Service 13 (c) by Raffael Marty Tuesday, July 6, 2010
  • 14. Normalization • Parsers “To analyze or separate (input, for example) into more easily processed components.” (answers.com) • Generate a common output format for vis-tools (e.g., CSV) • For example ‣ Regex /(d{1,3}.d{1,3}.d{1,3}.d{1,3})/g ‣ http://secviz.org/content/parser-exchange Logging as a Service 14 (c) by Raffael Marty Tuesday, July 6, 2010
  • 15. Visualize 15 Tuesday, July 6, 2010
  • 16. Choose Your Poison Logging as a Service 16 (c) by Raffael Marty Tuesday, July 6, 2010
  • 17. Reporting vs. Visualization • Reporting Libraries • Visualization Libraries - HighCharts - TheJIT - Flot - Graphael - Google Chart API - Protovis - Open Flash Chart - ProcessingJS - Flare JavaScript vs. Flash vs. XYZ Logging as a Service 17 (c) by Raffael Marty Tuesday, July 6, 2010
  • 18. HighCharts • Click-Through • On load - near real-time updates • AJAX data input via JSON • Zoom http://www.highcharts.com/ Logging as a Service 18 (c) by Raffael Marty Tuesday, July 6, 2010
  • 19. Google Visualization API http://code.google.com/apis/visualization/interactive_charts.html • JavaScript • Based on DataTables() • Many graphs • Playground - http://code.google.com/apis/ajax/playground Logging as a Service 19 (c) by Raffael Marty Tuesday, July 6, 2010
  • 20. ProtoVis • JavaScript based visualization library • Charting • Treemaps • BoxPlots • Parallel Coordinates • etc. http://vis.stanford.edu/protovis/ Logging as a Service 20 (c) by Raffael Marty Tuesday, July 6, 2010
  • 21. TheJIT http://thejit.org/ • JavaScript InfoVis Toolkit • Interactive • Link Graphs Logging as a Service 21 (c) by Raffael Marty Tuesday, July 6, 2010
  • 22. Processing •Visualization library •Java based •Interactive (event handling) •Number of libraries to -draw in OpenGL -read XML files -write PDF files •Processing JS -JavaScript -HTML 5 Canvas http://processingjs.org/ -Web IDE http://processing.org/ Logging as a Service 22 (c) by Raffael Marty Tuesday, July 6, 2010
  • 23. Building Your Own 23 Tuesday, July 6, 2010
  • 24. Build Your Own AfterGlow Loggly Regexes Google Vis Logging as a Service 24 (c) by Raffael Marty Tuesday, July 6, 2010
  • 25. Data Collection in the Cloud 25 Tuesday, July 6, 2010
  • 26. The (public) Cloud What it is Types • multi-tenancy • SaaS - Software • elastic • PaaS - Platform • “infinite” resources • IaaS - Infrastructure • pay as you go Benefits • self provisioning • No installation • No elaborate configurations It’s not • No maintenance • private data center • Great scalability • virtualization • 7x24 availability Logging as a Service 26 (c) by Raffael Marty Tuesday, July 6, 2010
  • 27. LaaS - Logging as a Service • All your data in one place • Loggly manages your data (index, store, archive, etc.) • Extremely fast search across all your data • Data source agnostic (no parsers) • Data management • access control • data segregation • data overview and summaries • API access Logging as a Service 27 (c) by Raffael Marty Tuesday, July 6, 2010
  • 28. Loggly Architecture Loggly Data Sources Clients user interface mobile-166 My syslog Data collection API Data access Proxies Distributed Indexers and Search Machines indexing and processing Distributed data store Logging as a Service 28 (c) by Raffael Marty Tuesday, July 6, 2010
  • 29. Loggly APIs • URL format: http://wiki.loggly.com/api-documentation http://<subdomain>.loggly.com/api/<resource> • RESTful API HTTP Based - Access through: /api/<resource> •GET - read - JSON, XML, JSONP output •POST - create • Authentication •PUT - update - Basic auth •DELETE - delete - oAuth http://loggly.loggly.com/api/search/?q=error syslog to: User: guest / Password: loggly logs.loggly.com:514 Logging as a Service 29 (c) by Raffael Marty Tuesday, July 6, 2010
  • 30. Search http://[domain].loggly.com/api/search?q=404 { "data": [ { "indexed": "2010-07-03T17:17:38.909Z", "ip": "75.101.249.172", "text": "Oct 13 20:00:38.018152 rule 57/0(match): pass in on xl1: 195.141.69.45.1030 > 62.2.32.250.53: 34388 [1au] [|domain] (DF)", "inputname": "logglyweb", "timestamp": "2010-07-03 10:17:38" }, { "indexed": "2010-07-03T17:17:37.879Z", "ip": "75.101.249.172", "text": "Oct 13 20:00:38.115862 rule 57/0(match): pass in on xl1: 195.141.69.45.1030 > 192.134.0.49.53: 49962 [1au] [|domain] (DF)", "inputname": "logglyapp", "timestamp": "2010-07-03 10:17:37" }, ... Logging as a Service 30 (c) by Raffael Marty Tuesday, July 6, 2010
  • 31. Parser Oct 13 20:00:38.018152 rule 57/0(match): pass in on xl1: 195.141.69.45.1030 > 62.2.32.250.53: 34388 [1au][|domain] (DF) Raw Oct 13 20:00:38.115862 rule 57/0(match): pass in on xl1: 195.141.69.45.1030 > 192.134.0.49.53: 49962 [1au][|domain] (DF) Oct 13 20:00:38.157238 rule 57/0(match): pass in on xl1: 195.141.69.45.1030 > 194.25.2.133.53: 14434 [1au][|domain] (DF) (.*) rule ([-d]+/d+)(.*?): (pass|block) (in|out) on (w+): (d+.d+.d+.d+).?(d*) [<>] Regex / Parser (d+.d+.d+.d+).?(d*): (.*) Oct 13 20:00:38.018152,57/0,match,pass,in,xl1,195.141.69.45,1030,62.2.32.250,53,34388 [1au][|domain] (DF) Normalized Oct 13 20:00:38.115862,57/0,match,pass,in,xl1,195.141.69.45,1030,192.134.0.49,53,49962 [1au][|domain] (DF) (CSV) Oct 13 20:00:38.157238,57/0,match,pass,in,xl1,195.141.69.45,1030,194.25.2.133,53,14434 [1au][|domain] (DF) Logging as a Service 31 (c) by Raffael Marty Tuesday, July 6, 2010
  • 32. Visualize Parser AfterGlow Grapher CSV file Graph file digraph structs { graph [label="AfterGlow 1.5.8", fontsize=8]; node [shape=ellipse, style=filled, Configuration fontsize=10, width=1, height=1, fixedsize=true]; edge [len=1.6]; color.source=“green” if ($fields[0] ne “d”) "aaelenes" -> "Printing Resume" ; cluster.target=regex_replace("(d+).")."/8" "abbe" -> "Information Encryption" ; threshold.event=5 "aanna" -> "Patent Access" ; size.target=$fields[1] "aatharuv" -> "Ping" ; } http://afterglow.sf.net Logging as a Service 32 (c) by Raffael Marty Tuesday, July 6, 2010
  • 33. AfterGlow Cloud Grapher Loggly JSON CSV DOT Graph Logging as a Service 33 (c) by Raffael Marty Tuesday, July 6, 2010
  • 34. Google Vis • JSON to Graphs • DataTable - used among all charts • Interactivity through events Logging as a Service 34 (c) by Raffael Marty Tuesday, July 6, 2010
  • 35. <script type="text/javascript"> Google Vis Code google.load('visualization', '1', {'packages':['motionchart', 'table', 'annotatedtimeline']}); google.setOnLoadCallback(call); var trends = new Array(); function call() { l! a $.ajax({ url: "http://logdog.loggly.com/api/search/?q=404&facets=True&buckets=100", n type:'GET', dataType: 'jsonp', username: 'xxxxx', password: 'xxxxxx', io success: function(data) { trends = data.data drawChart(); c t n } u }); f } t function drawChart() { o var data = new google.visualization.DataTable(); n data.addColumn('string', 'Search'); data.addColumn('datetime', 'Date'); is data.addColumn('number', 'Count'); e data.addRows(trends); od var chart = new google.visualization.MotionChart(document.getElementById('chart_div')); c chart.draw(data, {width: 600, height:300, state:state}); is var view = new google.visualization.DataView(data); h view.setRows(view.getFilteredRows([{column: 1, minValue: new Date(2007, 0, 1)}])); T var table = new google.visualization.Table(document.getElementById('test_dataview')); table.draw(view, {sortColumn: 1}); var time = new google.visualization.AnnotatedTimeLine(document.getElementById('timeline')); time.draw(timedata, {displayAnnotations: true}); } </script> Logging as a Service 35 (c) by Raffael Marty Tuesday, July 6, 2010
  • 36. Visualization Use-Cases 36 Tuesday, July 6, 2010
  • 37. NetFlow Visualization • Treemap • Protovis.JS • Size: Amount • Brightness: Variance • Color: Sensor • Shows: Scans - bright spots • Thanks to Chris Horsley Logging as a Service 37 (c) by Raffael Marty Tuesday, July 6, 2010
  • 38. Firewall Treemap Logging as a Service 38 (c) by Raffael Marty Tuesday, July 6, 2010
  • 39. Firewall Log Port Source IP Destination IP Logging as a Service 39 (c) by Raffael Marty Tuesday, July 6, 2010
  • 40. Visualization Resources 40 Tuesday, July 6, 2010
  • 41. http://secviz.org Share, discuss, challenge, and learn about security visualization. • List: secviz.org/mailinglist • Twitter: @secviz Logging as a Service 41 (c) by Raffael Marty Tuesday, July 6, 2010
  • 42. Applied Security Visualization • Bridging the gap between security and visualization • Hands-on, end to end examples • Data processing and analysis Chapters • Visualization • Compliance • Data Sources • Insider Threat • From Data to Graphs • Visualization Tools Addison Wesley (August, 2008) • Perimeter Threat ISBN: 0321510100 Logging as a Service 42 (c) by Raffael Marty Tuesday, July 6, 2010
  • 43. Thank You! raffael.marty@loggly.com @zrlram 43 Tuesday, July 6, 2010