DAVIX - Data Analysis and Visualization Linux

•

3 likes•4,193 views

DAVIX, a live CD for data analysis and visualization, brings the most important free tools for data processing and visualization to your desk. There is no hassle with installing an operating system or struggle to build the necessary tools to get started with visualization. You can completely dedicate your time to data analysis.

Software Technology

Security. Analytics. Insight.2
Data Analysis and Visualization LInuX

Security. Analytics. Insight.3
• Live Linux CD based on Ubuntu 13.10 Desktop
• Collection of free tools for data processing & visualization
• Tools work out of the box
• No compilation or installation required
• Comes with documentation
• Quick start description for the most important tools
• Links to manuals and tutorials
What is DAVIX?

Security. Analytics. Insight.4
User-Interface and Menus
Capture

Process

Visualize
Services

Security. Analytics. Insight.5
Selection of Tools
Process VisualizeCapture
• AfterGlow
• R
• RStudio
• Scapy
• Wireshark
• Argus
• BroIDS
• Snort
• LogStash
• nfdump
• p0f
• nmap
• PADS
• Cytoscape
• Gephi
• GGobi
• GnuPlot
• GraphViz
• Maltego
• PicViz
• Tulip
• Treemap• rsyslog
• syslog-ng
Services

Security. Analytics. Insight.6
New Tools in DAVIX 2014
• FlowTag
• Google Earth
• LogStash 1.4
• Maltego
• PRADS
• R Studio
• Tele Trafﬁc Tapper
• dns-browse
• netsed
• nsm-console
• rsyslog
• tcpstat
Studio

Security. Analytics. Insight.7
git://secviz/davix/
• Install script for all the tools
• apt-installs
• Manual installs
• R and R packages, RStudio
• UI setup
• Hardening of image
• Tool configurations - making tools log into logstash, for example [ to come ]
git://secviz/davix/wiki
• Documentation and user guides
GIT repo - Fork - Contribute!

Security. Analytics. Insight.8
To come:
• VM image
• ISO image
How To Use It
Download:
 
https://github.com/secviz/davix/tree/master/
install/davix-install-all.sh
!
Then run:
!
bash ./davix-install-all.sh
calls
• davix-install-*.sh
• davix-config.sh

Security. Analytics. Insight.9
List of tools with links to online tool
wiki pages
!
Install guide to install on your own 
Ubuntu Desktop
PDF User Manual

Security. Analytics. Insight.10
https://github.com/secviz/davix/wiki/!
!
• Information on all the tools
• Open to your contributions
Wiki User Manual

Security. Analytics. Insight.11
DEMO
Afterglow
GephiMondrian

Security. Analytics. Insight.12
More information …
!
!
http://davix.secviz.org
• New version will be announced and published here!
• ISO and VM download locations

What's hot

Monitoring an entire application is not a simple task, but with the right tools it is not a hard task either. However, events like Black Friday can push your application to the limit, and even cause crashes. As the system is stressed, it generates a lot more logs, which may crash the monitoring system as well. In this talk I will walk through the best practices when using the Elastic Stack to centralize and monitor your logs. I will also share some tricks to help you with the huge increase of traffic typical in Black Fridays. Topics include: * monitoring architectures * optimal bulk size * distributing the load * index and shard size * optimizing disk IO Takeaway: best practices when building a monitoring system with the Elastic Stack, advanced tuning to optimize and increase event ingestion performance.

Managing your Black Friday Logs NDC Oslo

David Pilato

Surveiller une application complexe n'est pas une tâche aisée, mais avec les bons outils, ce n'est pas si sorcier. Néanmoins, des périodes fortes telles que les opérations de type "Black Friday" (Vendredi noir) ou période de Noël peuvent pousser votre application aux limites de ce qu'elle peut supporter, ou pire, la faire crasher. Parce que le système est fortement sollicité, il génère encore davantage de logs qui peuvent également mettre à mal votre système de supervision. Dans cette session, j'aborderai les bonnes pratiques d'utilisation de la suite Elastic pour centraliser et monitorer vos logs. Je partagerai également avec vous quelques trucs et astuces pour vous aider à passer sans souci vos Vendredis noirs ! Nous verrons : * Les architectures de monitoring * Trouver la taille optimale pour l'API _bulk * Distribuer la charge * Taille des index et des shards * Optimiser les E/S disque Vous ressortirez de la session avec : des bonnes pratiques pour bâtir son système de monitoring avec la suite Elastic, le tuning avancé pour optimiser les performances d'ingestion et de recherche.

Managing your black friday logs Voxxed Luxembourg

David Pilato

Design Patterns For Real Time Streaming Data Analytics

DataWorks Summit

Reproducible Science with Python

Andreas Schreiber

We recently released the Neo4j graph algorithms library. You can use these graph algorithms on your connected data to gain new insights more easily within Neo4j. You can use these graph analytics to improve results from your graph data, for example by focusing on particular communities or favoring popular entities. We developed this library as part of our effort to make it easier to use Neo4j for a wider variety of applications. Many users expressed interest in running graph algorithms directly on Neo4j without having to employ a secondary system. We also tuned these algorithms to be as efficient as possible in regards to resource utilization as well as streamlined for later management and debugging. In this session we'll look at some of these graph algorithms and the types of problems that you can use them for in your applications.

Practical Graph Algorithms with Neo4j

jexp

SplunkLive! Zürich 2014 Beginner Workshop: Getting started with Splunk

Georg Knon

Intro to Python Data Analysis in Wakari

Karissa Rae McKelvey

ClickHouse Analytical DBMS: Introduction and Case Studies, by Alexander Zaitsev

Altinity Ltd

44CON 2014: Using hadoop for malware, network, forensics and log analysis

Michael Boman

Elasticsearch is a cloud-ready, super scalable search engine which is gaining a lot of popularity lately. It is mostly known for being extremely easy to setup and integrate with any technology stack.In this talk we will introduce Elasticdearch, and start by looking at some of its basic capabilities. We will demonstrate how it can be used for document search and even log analytics for DevOps and distributed debugging, and peek into more advanced usages like the real-time aggregations and percolation. Obviously, we will make sure to demonstrate how Elasticsearch can be scaled out easily to work on a distributed architecture and handle pretty much any load.

Elasticsearch Distributed search & analytics on BigData made easy

Itamar

Druid @ branch

Biswajit Das

Splunk n-box-splunk conf-2017

Mohamad Hassan

Elasticsearch in Netflix

Danny Yuan

Detecting Hacks: Anomaly Detection on Networking Data

DataWorks Summit

HDP2 and YARN operations point

Treasure Data, Inc.

Scaling Self Service Analytics with Databricks and Apache Spark with Amelia C...

Databricks

Streaming Aggregation in Solr - New Horizons for Search: Presented by Erick E...

Lucidworks

Presenter: Antonio Alcocer, Big Data Architect at Stratio Telefonica is the incumbent telecommunications network operator in Spain and the fourth one in capitalisation in the world. Cyber security is one of our most successful businesses worldwide. We provide monitoring and protecting clients from attacks. We analyze millions of data from multiple sources including social media, DNS records, and underground internet, to generate alerts and security reports for our clients. This use case required a Big Data component capable of processing the data and extract its information in real-time; warnings and alerts are time-sensitive in order to deal efficiently with security attacks. Our original architecture was the typical one used for data fusion systems. It included several collectors, a processing layer based on legacy systems, and a data store. The initial setup included a MongoDB database and an ad-hoc application. This solution however proved to be unfit for the specific purpose of dispatching alerts. We proposed to use Cassandra and Spark instead. This approach did manage to fulfill our original specifications as intended. Our talk will explain the reasons why we migrated the architecture and how the adopted solution based on Spark and Cassandra solved our problem.

Cassandra Summit 2014: Apache Cassandra at Telefonica CBS

DataStax Academy

Apache Eagle: eBay构建开源分布式实时预警引擎实践

Hao Chen

Building a Scalable Distributed Stats Infrastructure with Storm and KairosDB Many startups collect and display stats and other time-series data for their users. A supposedly-simple NoSQL option such as MongoDB is often chosen to get started... which soon becomes 50 distributed replica sets as volume increases. This talk describes how we designed a scalable distributed stats infrastructure from the ground up. KairosDB, a rewrite of OpenTSDB built on top of Cassandra, provides a solid foundation for storing time-series data. Unfortunately, though, it has some limitations: millisecond time granularity and lack of atomic upsert operations which make counting (critical to any stats infrastructure) a challenge. Additionally, running KairosDB atop Cassandra inside AWS brings its own set of challenges, such as managing Cassandra seeds and AWS security groups as you grow or shrink your Cassandra ring. In this deep-dive talk, we explore how we've used a mix of open-source and in-house tools to tackle these challenges and build a robust, scalable, distributed stats infrastructure.

Building a Scalable Distributed Stats Infrastructure with Storm and KairosDB

Cody Ray

What's hot (20)

Managing your Black Friday Logs NDC Oslo

Managing your black friday logs Voxxed Luxembourg

Design Patterns For Real Time Streaming Data Analytics

Reproducible Science with Python

Practical Graph Algorithms with Neo4j

SplunkLive! Zürich 2014 Beginner Workshop: Getting started with Splunk

Intro to Python Data Analysis in Wakari

ClickHouse Analytical DBMS: Introduction and Case Studies, by Alexander Zaitsev

44CON 2014: Using hadoop for malware, network, forensics and log analysis

Elasticsearch Distributed search & analytics on BigData made easy

Druid @ branch

Splunk n-box-splunk conf-2017

Elasticsearch in Netflix

Detecting Hacks: Anomaly Detection on Networking Data

HDP2 and YARN operations point

Scaling Self Service Analytics with Databricks and Apache Spark with Amelia C...

Streaming Aggregation in Solr - New Horizons for Search: Presented by Erick E...

Cassandra Summit 2014: Apache Cassandra at Telefonica CBS

Apache Eagle: eBay构建开源分布式实时预警引擎实践

Building a Scalable Distributed Stats Infrastructure with Storm and KairosDB

Viewers also liked

Supercharging Visualization with Data Mining

Raffael Marty

RSA 2006 - Visual Security Event Analysis

Raffael Marty

The security industry is talking a lot about threat intelligence; external information that a company can leverage to understand where potential threats are knocking on the door and might have already perpetrated the network boundaries. Conversations with many CERTs have shown that we have to stop relying on knowledge about how attacks have been conducted in the past and start ‘hunting’ for signs of compromises and anomalies in our own environments. In this presentation we explore how the decade old field of security visualization has emerged. We show how we have applied advanced analytics and visualization to create our own threat intelligence and investigated lateral movement in a Fortune 50 company. Visualization. Data science. No machine learning. But pretty pictures.What is internal threat intelligence? Check out http://www.darkreading.com/analytics/creating-your-own-threat-intel-through-hunting-and-visualization/a/d-id/1321225

Creating Your Own Threat Intel Through Hunting & Visualization

Raffael Marty

Offloading BPF Implementation to FPGA-NIC したいねって話

slankdev

サイボウズ・ラボユース成果報告会

slankdev

企業向けクラウドサービスの開発・運用　悩みどころのパターンと対策

Ryo Mitoma

サイボウズ・ラボ成果発表会

Komei Kamiya

High Performance Networking with DPDK & Multi/Many Core

slankdev

Viewers also liked (8)

Supercharging Visualization with Data Mining

RSA 2006 - Visual Security Event Analysis

Creating Your Own Threat Intel Through Hunting & Visualization

Offloading BPF Implementation to FPGA-NIC したいねって話

サイボウズ・ラボユース成果報告会

企業向けクラウドサービスの開発・運用　悩みどころのパターンと対策

サイボウズ・ラボ成果発表会

High Performance Networking with DPDK & Multi/Many Core

Similar to DAVIX - Data Analysis and Visualization Linux

DAVIX - VizSec 2008

Raffael Marty

Zabbix introduction ( RadixCloud Radix Technologies SA)

Martin Markovski

"In love with Open Source : Past, Present and Future" : Keynote OSDConf 2014

Piyush Kumar

Infosecurity.be 2019: What are relevant open source security tools you should...

B.A.

Rootconf 2017 - State of the Open Source monitoring landscape

NETWAYS

Sydney Drupal News May 2012

Ryan Cross

GraphTour - Neo4j Platform Overview

Neo4j

OWASP 2015 AppSec EU ZAP 2.4.0 and beyond..

Simon Bennetts

Ross King, Project Director of SCAPE, gave a short presentation of the EU funded project SCAPE, including descriptions of tools for planning and monitoring digital preservation, scalable computation and repositories, SCAPE Testbeds and where to learn more. The presentation was given at the workshop ‘Preservation at Scale’ http://bit.ly/17ppAln in connection with the iPres2013 conference in Lissabon, Portugal, in September 2013.

SCAPE - Scalable Preservation Environments

SCAPE Project

JoinSEC 2013 London - ZAP Intro

Simon Bennetts

Azure and deep learning

David Giard

DevSecOps Beginners Guide : How to secure process in DevOps with OpenSource

DevOps Indonesia

Data mining tools (R , WEKA, RAPID MINER, ORANGE)

Krishna Petrochemicals

Hashicorp Products Overview

Uchit Vyas ☁

Lesson_08_Continuous_Monitoring.pdf

Minh Quân Đoàn

CodeMotion Amsterdam 2018 - Microservices in action at the Dutch National Police

Bert Jan Schrijver

At the Cloud, Big Data and Internet division of the Dutch National Police, 4 DevOps teams use the latest open source technology to build high tech, cloud native web applications using Spring Boot, Angular 5, Spark, Kafka and Jenkins 2. I'll share our experiences and real-world use cases for microservices. I’ll show how 4 teams work together on one product and I’ll talk about how we apply the principles of DevOps and Continuous Delivery. I’ll show how we handle security, build pipelines, test automation, performance tests, service discovery, automated deployments, monitoring and more!

Microservices in action at the Dutch National Police - Bert Jan Schrijver - C...

Codemotion

Grafana.pptx

Bhushan Rane

This presentation was from Joomla day 2016 held right here in KLCC Malaysia. Astiostech presented several important factors to consider when monitoring a web service with of course special focus on Joomla. However, these guidelines can be used for just about any web service you may want to monitor. Monitoring is pivotal to a web infrastructure and it should not be considered today as a luxury. With tools like Nagios XI, we can simply start monitoring with mere clicks of a web browser and you're pretty much on the right track.

Functionality, security and performance monitoring of web assets (e.g. Joomla...

Sanjay Willie

OWASP 2013 EU Tour Amsterdam ZAP Intro

Simon Bennetts

Similar to DAVIX - Data Analysis and Visualization Linux (20)

DAVIX - VizSec 2008

Zabbix introduction ( RadixCloud Radix Technologies SA)

"In love with Open Source : Past, Present and Future" : Keynote OSDConf 2014

Infosecurity.be 2019: What are relevant open source security tools you should...

Rootconf 2017 - State of the Open Source monitoring landscape

Sydney Drupal News May 2012

GraphTour - Neo4j Platform Overview

OWASP 2015 AppSec EU ZAP 2.4.0 and beyond..

SCAPE - Scalable Preservation Environments

JoinSEC 2013 London - ZAP Intro

Azure and deep learning

DevSecOps Beginners Guide : How to secure process in DevOps with OpenSource

Data mining tools (R , WEKA, RAPID MINER, ORANGE)

Hashicorp Products Overview

Lesson_08_Continuous_Monitoring.pdf

CodeMotion Amsterdam 2018 - Microservices in action at the Dutch National Police

Microservices in action at the Dutch National Police - Bert Jan Schrijver - C...

Grafana.pptx

Functionality, security and performance monitoring of web assets (e.g. Joomla...

OWASP 2013 EU Tour Amsterdam ZAP Intro

More from Raffael Marty

Exploring the Defender's Advantage

Raffael Marty

Extended Detection and Response, or XDR for short, is one of the acronyms that are increasingly used by cybersecurity vendors to explain their approach to solving the cyber security problem. We have been spending trillions of dollars on approaches to secure our systems and data, with what success? Cybersecurity is still one of the biggest and most challenging areas that companies, small and large, are dealing with. XDR is another approach driven by security vendors to solve this problem. The challenge is that every vendor defines XDR slightly differently and makes it fit their own “challenge du jour” for marketing and selling their products. In this presentation we will demystify the XDR acronym and put a working model behind it. Together, we will explore why XDR is a fabulous concept, but also discover that it’s nothing revolutionarily new. With an MSP lens, we will explore what the XDR benefits are for small and medium businesses and what it means to the security strategy of both MSPs and their clients. The audience will leave with a clear understanding of what XDR is, how the technology matters to them, and how XDR will ultimately help them secure their customers and enable trusted commerce.

Extended Detection and Response (XDR)An Overhyped Product Category With Ulti...

Raffael Marty

Blog Post: http://raffy.ch/blog. - Video: https://youtu.be/nk5uz0VZrxM In this video we talk about the world of security data or log data. In the first section, we dive into a bit of a history lesson around log management, SIEM, and big data in security. We then shift to the present to discuss some of the challenges that we face today with managing all of that data and also discuss some of the trends in the security analytics space. In the third section, we focus on the future. What does tomorrow hold in the SIEM / security data space? What are some of the key features we will see and how does this matter to the user of these approaches.

How To Drive Value with Security Data

Raffael Marty

The cyber security industry has spent trillions of dollars to keep external attackers at bay. To what effect? We still don't see an end to the cat and mouse game between attackers and the security industry; zero day attacks, new vulnerabilities, ever increasingly sophisticated attacks, etc. We need a paradigm shift in security. A shift away from traditional threat intelligence and indicators of compromise (IOCs). We need to look at understanding behaviors. Those of devices and those of humans. What are the security approaches and trends that will make an actual difference in protecting our critical data and intellectual property; not just from external attackers, but also from malicious insiders? We will explore topics from the 'all solving' artificial intelligence to risk-based security. We will look at what is happening within the security industry itself, where startups are putting placing their bets, and how human factors will play an increasingly important role in security, along with all of the potential challenges that will create.

Cyber Security Beyond 2020 – Will We Learn From Our Mistakes?

Raffael Marty

Companies have AI projects. Security products use AI to keep attackers out and insiders at bay. But what is this "AI" that everyone talks about? In this talk we will explore what artificial intelligence in cyber security is, where the limitations and dangers are, and in what areas we should invest more in AI. We will talk about some of the recent failures of AI in security and invite a conversation about how we verify artificially intelligent systems to understand how much trust we can place in them. Alongside the AI conversation, we will discover that we need to make a shift in our traditional approach to cyber security. We need to augment our reactive approaches of studying adversary behaviors to understanding behaviors of users and machines to inform a risk-driven approach to security that prevents even zero day attacks.

Artificial Intelligence – Time Bomb or The Promised Land?

Raffael Marty

In this presentation I explore the topic of artificial intelligence in cyber security. What is AI and how do we get to real intelligence in a cyber context. I outline some of the dangers of the way we are using algorithms (AI, ML) today and what that leads to. We then explore how we can add real intelligence through export knowledge to the problem of finding attackers and anomalies in our applications and networks. Presented at AI 4 Cyber in NYC on April 30, 2019

Understanding the "Intelligence" in AI

Raffael Marty

Security Chat 5.0

Raffael Marty

Link to the video of the presentation: https://www.youtube.com/watch?v=WG1k-Xh1TqM Every single security company is talking in some way or another about how they are applying machine learning. Companies go out of their way to make sure they mention machine learning and not statistics when they explain how they work. Recently, that's not enough anymore either. As a security company you have to claim artificial intelligence to be even part of the conversation. Guess what. It's all baloney. We have entered a state in cyber security that is, in fact, dangerous. We are blindly relying on algorithms to do the right thing. We are letting deep learning algorithms detect anomalies in our data without having a clue what that algorithm just did. In academia, they call this the lack of explainability and verifiability. But rather than building systems with actual security knowledge, companies are using algorithms that nobody understands and in turn discover wrong insights. In this talk, I will show the limitations of machine learning, outline the issues of explainability, and show where deep learning should never be applied. I will show examples of how the blind application of algorithms (including deep learning) actually leads to wrong results. Algorithms are dangerous. We need to revert back to experts and invest in systems that learn from, and absorb the knowledge, of experts.

AI & ML in Cyber Security - Why Algorithms are Dangerous

Raffael Marty

Every single security company is talking in some way or another about how they are applying machine learning. Companies go out of their way to make sure they mention machine learning and not statistics when they explain how they work. Recently, that's not enough anymore either. As a security company you have to claim artificial intelligence to be even part of the conversation. Guess what. It's all baloney. We have entered a state in cyber security that is, in fact, dangerous. We are blindly relying on algorithms to do the right thing. We are letting deep learning algorithms detect anomalies in our data without having a clue what that algorithm just did. In academia, they call this the lack of explainability and verifiability. But rather than building systems with actual security knowledge, companies are using algorithms that nobody understands and in turn discover wrong insights. In this talk I will show the limitations of machine learning, outline the issues of explainability, and show where deep learning should never be applied. I will show examples of how the blind application of algorithms (including deep learning) actually leads to wrong results. Algorithms are dangerous. We need to revert back to experts and invest in systems that learn from, and absorb the knowledge, of experts.

AI & ML in Cyber Security - Why Algorithms Are Dangerous

Raffael Marty

It's an interesting exercise to look back to the year 2000 to see how we approached cyber security. We just started to realize that data might be a useful currency, but for the most part, security pursued preventative avenues, such as firewalls, intrusion prevention systems, and anti-virus. With the advent of log management and security incident and event management (SIEM) solutions we started to gather gigabytes of sensor data and correlate data from different sensors to improve on their weaknesses and accelerate their strengths. But fundamentally, such solutions didn't scale that well and struggled to deliver real security insight. Today, cybersecurity wouldn't work anymore without large scale data analytics and machine learning approaches, especially in the realm of malware classification and threat intelligence. Nonetheless, we are still just scratching the surface and learning where the real challenges are in data analytics for security. This talk will go on a journey of big data in cybersecurity, exploring where big data has been and where it must go to make a true difference. We will look at the potential of data mining, machine learning, and artificial intelligence, as well as the boundaries of these approaches. We will also look at both the shortcomings and potential of data visualization and the human computer interface. It is critical that today's systems take into account the human expert and, most importantly, provide the right data.

Delivering Security Insights with Data Analytics and Visualization

Raffael Marty

We are writing the year 2017. Cyber security has been a discipline for many years and thousands of security companies are offering solutions to deter and block malicious actors in order to keep our businesses operating and our data confidential. But fundamentally, cyber security has not changed during the last two decades. We are still running Snort and Bro. Firewalls are fundamentally still the same. People get hacked for their poor passwords and we collect logs that we don't know what to do with. In this talk I will paint a slightly provocative and dark picture of security. Fundamentally, nothing has really changed. We'll have a look at machine learning and artificial intelligence and see how those techniques are used today. Do they have the potential to change anything? How will the future look with those technologies? I will show some practical examples of machine learning and motivate that simpler approaches generally win. Maybe we find some hope in visualization? Or maybe Augmented reality? We still have a ways to go.

AI & ML in Cyber Security - Welcome Back to 1999 - Security Hasn't Changed

Raffael Marty

Ensuring security of a company’s data and infrastructure has largely become a data analytics challenge. It is about finding and understanding patterns and behaviors that are indicative of malicious activities or deviations from the norm. Data, Analytics, and Visualization are used to gain insights and discover those malicious activities. These three components play off of each other, but also have their inherent challenges. A few examples will be given to explore and illustrate some of these challenges,

Security Insights at Scale

Raffael Marty

The security industry is talking a lot about threat intelligence; external information that a company can leverage to understand where potential threats are knocking on the door and might have already perpetrated the network boundaries. Conversations with many CERTs have shown that we have to stop relying on knowledge about how attacks have been conducted in the past and start 'hunting' for signs of compromises and anomalies in our own environments. In this presentation we explore how the decade old field of security visualization has emerged. We show how we have applied advanced analytics and visualization to create our own threat intelligence and investigated lateral movement in a Fortune 50 company. Visualization. Data science. No machine learning. But pretty pictures. Here is a blog post I wrote a bit ago about the general theme of internal threat intelligence: http://www.darkreading.com/analytics/creating-your-own-threat-intel-through-hunting-and-visualization/a/d-id/1321225?

Creating Your Own Threat Intel Through Hunting & Visualization

Raffael Marty

The extent and impact of recent security breaches is showing that current security approaches are just not working. But what can we do to protect our business? We have been advocating monitoring for a long time as a way to detect subtle, advanced attacks that are still making it through our defenses. However, products have failed to deliver on this promise. Current solutions don't scale in both data volume and analytical insights. In this presentation we will explore what security monitoring is. Specifically, we are going to explore the question of how to visualize a billion log records. A number of security visualization examples will illustrate some of the challenges with big data visualization. They will also help illustrate how data mining and user experience design help us get a handle on the security visualization challenges - enabling us to gain deep insight for a number of security use-cases.

Visualization in the Age of Big Data

Raffael Marty

Big Data Visualization

Raffael Marty

The extent and impact of recent security breaches is showing that current approaches are just not working. But what can we do to protect our business? We have been advocating monitoring for a long time as a way to detect subtle, advanced attacks. However, products have failed to deliver on this promise. Current solutions don't scale in both data volume and analytical insights. In this presentation we will explore why it is so hard to come up with a security monitoring (or shall we call it security intelligence) approach that helps find sophisticated attackers in all the data collected. We are going to explore the question of how to visualize a billion events. We are going to look at a number of security visualization examples to illustrate the problem and some possible solutions. These examples will also help illustrate how data mining and user experience design help us get a handle of the security visualization challenges - enabling us to gain deep insight for a number of security use-cases.

The Heatmap  - Why is Security Visualization so Hard?

Raffael Marty

Vision is a human’s dominant sense. It is the communication channel with the highest bandwidth into the human brain. Security tools and applications need to make better use of information visualization to enhance human computer interactions and information exchange. In this talk we will explore a few basic principles of information visualization to see how they apply to cyber security. We will explore both visualization as a data presentation, as well as a data discovery tool. We will address questions like: What makes for effective visualizations? What are some core principles to follow when designing a dashboard? How do you go about visually exploring a terabyte of data? And what role do big data and data mining play in security visualization? The presentation is filled with visualizations of security data to help translate the theoretical concepts into tangible applications.

Visualization for Security

Raffael Marty

This presentation explores why it is so hard to come up with a security monitoring (or shall we call it security intelligence) approach that helps find sophisticated attackers in all the data collected. It explores the question of how to visualize a billion events. To do so, the presentation dives deeply into heatmaps - matrices - as an example of a simple type of visualization. While these heatmaps are very simple, they are incredibly versatile and help us think about the problem of security visualization. They help illustrate how data mining and user experience design help get a handle of the security visualization challenges - enabling us to gain deep insight for a number of security use-cases.

The Heatmap  - Why is Security Visualization so Hard?

Raffael Marty

Cloud - Security - Big Data

Raffael Marty

Video can be found at: http://youtu.be/CEAMF0TaUUU In the Cyber Security domain, we have been collecting ‘big data’ for almost two decades. The volume and variety of our data is extremely large, but understanding and capturing the semantics of the data is even more of a challenge. Finding the needle in the proverbial haystack has been attempted from many different angles. In this talk we will have a look at what approaches have been explored, what has worked, and what has not. We will see that there is still a large amount of work to be done and data mining is going to play a central role. We’ll try to motivate that in order to successfully find bad guys, we will have to embrace a solution that not only leverages clever data mining, but employs the right mix between human computer interfaces, data mining, and scalable data platforms.

Cyber Security – How Visual Analytics Unlock Insight

Raffael Marty

More from Raffael Marty (20)

Exploring the Defender's Advantage

Extended Detection and Response (XDR)An Overhyped Product Category With Ulti...

How To Drive Value with Security Data

Cyber Security Beyond 2020 – Will We Learn From Our Mistakes?

Artificial Intelligence – Time Bomb or The Promised Land?

Understanding the "Intelligence" in AI

Security Chat 5.0

AI & ML in Cyber Security - Why Algorithms are Dangerous

AI & ML in Cyber Security - Why Algorithms Are Dangerous

Delivering Security Insights with Data Analytics and Visualization

AI & ML in Cyber Security - Welcome Back to 1999 - Security Hasn't Changed

Security Insights at Scale

Creating Your Own Threat Intel Through Hunting & Visualization

Visualization in the Age of Big Data

Big Data Visualization

The Heatmap  - Why is Security Visualization so Hard?

Visualization for Security

The Heatmap  - Why is Security Visualization so Hard?

Cloud - Security - Big Data

Cyber Security – How Visual Analytics Unlock Insight

Recently uploaded

WSO2CON 2024 - Navigating API Complexity: REST, GraphQL, gRPC, Websocket, Web...

WSO2

WSO2CON 2024 - Designing Event-Driven Enterprises: Stories of Transformation

WSO2

WSO2CON 2024 - Building the API First Enterprise – Running an API Program, fr...

WSO2

ADR, or Architecture Decision Record, is a valuable tool in software development for several reasons. It provides a centralized location for documenting and tracking architectural decisions, aiding both current and future team members. ADRs enhance communication among team members by documenting the rationale behind architectural decisions, especially beneficial during onboarding of new team members or when revisiting decisions. They serve as a knowledge base, enabling teams to learn from past decisions and refine their decision-making process. Additionally, ADRs contribute to transparency by helping stakeholders understand the reasons behind specific architectural choices. As with any other tool or process, introducing them into an organization can face several obstacles, and overcoming these challenges is crucial for successful implementation. In this talk I go through some common problems and our way of solving them.

Architecture decision records - How not to get lost in the past

Papp Krisztián

Announcing Codolex 2.0 from GDK Software

Jim McKeeth

Artyushina_Guest lecture_YorkU CS May 2024.pptx

AnnaArtyushina1

WSO2Con2024 - Unleashing the Financial Potential of 13 Million People

WSO2

WSO2CON 2024 - Not Just Microservices: Rightsize Your Services!

WSO2

Evolving Data Governance for the Real-time Streaming and AI Era

confluent

WSO2Con2024 - Low-Code Integration Tooling

WSO2

WSO2Con2024 - Navigating the Digital Landscape: Transforming Healthcare with ...

WSO2

Azure Native Qumulo scales elastically for common High Performance Compute (HPC) workloads based on application requirements for: Financial Services, Automotive, Genomics / Life Sciences, Media and Entertainment, Energy, Oil and Gas, and more. Performance can be increased (and elastically decreased) much higher than the examples shown here. These slides offer a glimpse into ANQ's HPC capabilities, although at a smaller scale. We invite YOU to do your own testing (with a free ANQ trial) and work with us to test your HPC workloads in Azure.

AzureNativeQumulo_HPC_Cloud_Native_Benchmarks.pdf

ryanfarris8

WSO2Con2024 - Enabling Transactional System's Exponential Growth With Simplicity

WSO2

WSO2CON 2024 - How to Run a Security Program

WSO2

WSO2Con2024 - Organization Management: The Revolution in B2B CIAM

WSO2

WSO2Con2024 - Facilitating Broadband Switching Services for UK Telecoms Provi...

WSO2

WSO2CON 2024 Slides - Open Source to SaaS

WSO2

WSO2Con2024 - Software Delivery in Hybrid Environments

WSO2

WSO2CON 2024 - How CSI Piemonte Is Apifying the Public Administration

WSO2

WSO2CON 2024 - Software Engineering for Digital Businesses

WSO2

Recently uploaded (20)

WSO2CON 2024 - Navigating API Complexity: REST, GraphQL, gRPC, Websocket, Web...

WSO2CON 2024 - Designing Event-Driven Enterprises: Stories of Transformation

WSO2CON 2024 - Building the API First Enterprise – Running an API Program, fr...

Architecture decision records - How not to get lost in the past

Announcing Codolex 2.0 from GDK Software

Artyushina_Guest lecture_YorkU CS May 2024.pptx

WSO2Con2024 - Unleashing the Financial Potential of 13 Million People

WSO2CON 2024 - Not Just Microservices: Rightsize Your Services!

Evolving Data Governance for the Real-time Streaming and AI Era

WSO2Con2024 - Low-Code Integration Tooling

WSO2Con2024 - Navigating the Digital Landscape: Transforming Healthcare with ...

AzureNativeQumulo_HPC_Cloud_Native_Benchmarks.pdf

WSO2Con2024 - Enabling Transactional System's Exponential Growth With Simplicity

WSO2CON 2024 - How to Run a Security Program

WSO2Con2024 - Organization Management: The Revolution in B2B CIAM

WSO2Con2024 - Facilitating Broadband Switching Services for UK Telecoms Provi...

WSO2CON 2024 Slides - Open Source to SaaS

WSO2Con2024 - Software Delivery in Hybrid Environments

WSO2CON 2024 - How CSI Piemonte Is Apifying the Public Administration

WSO2CON 2024 - Software Engineering for Digital Businesses

DAVIX - Data Analysis and Visualization Linux

1. HoneyNet Workshop, Warsaw May 13, 2014

2. Security. Analytics. Insight.2 Data Analysis and Visualization LInuX 

3. Security. Analytics. Insight.3 • Live Linux CD based on Ubuntu 13.10 Desktop • Collection of free tools for data processing & visualization • Tools work out of the box • No compilation or installation required • Comes with documentation • Quick start description for the most important tools • Links to manuals and tutorials What is DAVIX?

4. Security. Analytics. Insight.4 User-Interface and Menus Capture Process Visualize Services

5. Security. Analytics. Insight.5 Selection of Tools Process VisualizeCapture • AfterGlow • R • RStudio • Scapy • Wireshark • Argus • BroIDS • Snort • LogStash • nfdump • p0f • nmap • PADS • Cytoscape • Gephi • GGobi • GnuPlot • GraphViz • Maltego • PicViz • Tulip • Treemap• rsyslog • syslog-ng Services

6. Security. Analytics. Insight.6 New Tools in DAVIX 2014 • FlowTag • Google Earth • LogStash 1.4 • Maltego • PRADS • R Studio • Tele Trafﬁc Tapper • dns-browse • netsed • nsm-console • rsyslog • tcpstat Studio

7. Security. Analytics. Insight.7 git://secviz/davix/ • Install script for all the tools • apt-installs • Manual installs • R and R packages, RStudio • UI setup • Hardening of image • Tool configurations - making tools log into logstash, for example [ to come ] git://secviz/davix/wiki • Documentation and user guides GIT repo - Fork - Contribute!

8. Security. Analytics. Insight.8 To come: • VM image • ISO image How To Use It Download:   https://github.com/secviz/davix/tree/master/ install/davix-install-all.sh ! Then run: ! bash ./davix-install-all.sh calls • davix-install-*.sh • davix-config.sh

9. Security. Analytics. Insight.9 List of tools with links to online tool wiki pages ! Install guide to install on your own  Ubuntu Desktop PDF User Manual

10. Security. Analytics. Insight.10 https://github.com/secviz/davix/wiki/! ! • Information on all the tools • Open to your contributions Wiki User Manual

11. Security. Analytics. Insight.11 DEMO Afterglow GephiMondrian

12. Security. Analytics. Insight.12 More information … ! ! http://davix.secviz.org • New version will be announced and published here! • ISO and VM download locations

13. 13 raffael.marty@pixlcloud.com

DAVIX - Data Analysis and Visualization Linux

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (8)

Similar to DAVIX - Data Analysis and Visualization Linux

Similar to DAVIX - Data Analysis and Visualization Linux (20)

More from Raffael Marty

More from Raffael Marty (20)

Recently uploaded

Recently uploaded (20)

DAVIX - Data Analysis and Visualization Linux