Big Data Visualization

Raffael Marty, CEO Big Data Visualization London February, 2015

Security. Analytics. Insight.2 • Visualization • Design Principles • Dashboards • SOC Dashboard • Data Discovery and Exploration • Data Requirements for Visualization • Big Data Lake Overview

Security. Analytics. Insight.3 I am Raffy - I do Viz! IBM Research

4 Visualization

Security. Analytics. Insight.5 Why Visualization? the stats ... http://en.wikipedia.org/wiki/Anscombe%27s_quartet the data...

Security. Analytics. Insight.6 Why Visualization? http://en.wikipedia.org/wiki/Anscombe%27s_quartet Human analyst: • patterndetection • remembers context • fantasticintuition • canpredict

Security. Analytics. Insight.7 Visualization To … Present / Communicate Discover / Explore

Design Principles

Security. Analytics. Insight.9 Choosing Visualizations Objective AudienceData

Security. Analytics. Insight.10 • Objective: Find attackers in the network moving laterally • Defines data needed (netflow, sflow, …) • maybe restrict to a network segment • Audience: security analyst, risk team, … • Informs how to visualize / present data For Example - Lateral Movement Recon Weaponize Deliver Exploit Install C2 Act

Security. Analytics. Insight.11 • Show comparisons, contrasts, differences • Show causality, mechanism, explanation, systematic structure. • Show multivariate data; that is, show more than 1 or 2 variables. by Edward Tufte Principals of Analytic Design

Security. Analytics. Insight.12 Show Context 42

Security. Analytics. Insight. 42 is just a number and means nothing without context 13 Show Context

Security. Analytics. Insight.15 Use Numbers To Highlight Most Important Parts of Data Numbers Summaries

Security. Analytics. Insight.16 Additional information about objects, such as: • machine • roles • criticality • location • owner • … • user • roles • ofﬁce location • … Add Context source destination machine and   user context machine role user role

Security. Analytics. Insight.17 Trafﬁc Flow Analysis With Context

Security. Analytics. Insight.18 http://www.sciﬁinterfaces.com/ • Black background • Blue or green colors • Glow Aesthetics Matter

Security. Analytics. Insight.19 B O R I N G

Security. Analytics. Insight.20 Sexier

Security. Analytics. Insight.21 • Audience, audience, audience! • Comprehensive Information (enough context) • Highlight important data • Use graphics when appropriate • Good choice of graphics and design • Aesthetically pleasing • Enough information to decide if action is necessary • No scrolling • Real-time vs. batch? (Refresh-rates) • Clear organization Dashboard Design Principles

22 SOC Dashboards

Security. Analytics. Insight.23 Mostly Blank

Security. Analytics. Insight.24 • Disappears too quickly • Analysts focus is on their own screens • SOC dashboard just distracts • Detailed information not legible • Put the detailed dashboards on the analysts screens! Dashboards For Discovery

Security. Analytics. Insight.25 • Provide analyst with context • “What else is going on in the environment right now?” • Bring Into Focus • Turn something benign into something interesting • Disprove • Turn something interesting into something benign Use SOC Dashboard For Context Environment informs detection policies

Security. Analytics. Insight.26 Show Comparisons Current Measure week prior

Security. Analytics. Insight.27 • News feed summary (FS ISAC feeds, mailinglists, threat feeds) • Monitoring twitter or IRC for certain activity / keywords • Volumes or metrics (e.g., #firewall blocks, #IDS alerts, #failed transactions) • Top N metrics: • Top 10 suspicious users • Top 10 servers connecting outbound What To Put on Screens Provide context to individual security alerts http://raffy.ch/blog/2015/01/15/dashboards-in-the-security-opartions-center-soc/

28 Data Discovery & Exploration

Security. Analytics. Insight.29 Visualize Me Lots (>1TB) of Data

Security. Analytics. Insight.30 Information Visualization Mantra Overview Zoom / Filter Details on Demand Principle by Ben Shneiderman • summary / aggregation • data mining • signal detection (IDS, behavioral, etc.)

Security. Analytics. Insight.31 • Access to data • Parsed data and data context • Data architecture for central data access and fast queries • Application of data mining (how?, what?, scalable, …) • Visualization tools that support • Complex visual types (||-coordinates, treemaps,   heat maps, link graphs) • Linked views • Data mining (clustering, …) • Collaboration, information sharing • Visual analytics workﬂow Visualization Challenges

Big Data Lake

Security. Analytics. Insight.33 • One central location to store all cyber security data • “Data collected only once and third party software leveraging it” • Scalability and interoperability • More than deploying an off the shelf product from a vendor • Data use inﬂuences both data formats and technologies to store the data • search, analytics, relationships, and distributed processing • correlation, and statistical summarization • What to do with Context? Enrich or join? • Hard problems: • Parsing: can you re-parse? Common naming scheme! • Data store capabilities (search, analytics, distributed processing, etc.) • Access to data: SQL (even in Hadoop context), how can products access the data? The Big Data Lake

Security. Analytics. Insight.34 Federated Data Access SIEM dispatcher SIEM   connector SIEM console Prod A AD / LDAP HR … IDS FW Prod B DBs Data Lake Caveats: • Dispatcher? • Standard access to dispatcher /  products enabled • Data lake technology? SNMP

Security. Analytics. Insight.35 Multiple Data Stores raw logs key-value structured real-time  processing (un)-structured data context SQL s t o r a g e stats index queue distributed  processing a c c e s s graph Caveat: • Need multiple types of   data stores

Security. Analytics. Insight.36 Technologies (Example) raw logs key-value (Cassandra) columnar (parquet) real-time  processing (Spark) (un)-structured data context SQL (Impala, SparkSQL) H D F S aggregates index (ES) queue (Kafka) distributed  processing (Spark) a c c e s s graph (GraphX) Caveat: • No out of the box solution available

Security. Analytics. Insight.37 SIEM Integration - Log Management First SIEM columnar or search engine  or log management processing SIEM   connector raw logs SIEM console SQL or search  interface processing ﬁltering H D F S e.g., PIG parsing

Security. Analytics. Insight.38 Simple SIEM Integration raw, csv, json ﬂume log data SQL (Impala, with SerDe) H D F S SIEM   connector SIEM Requirement: • SIEM connector to forward text- based data to Flume. SQL interface Tableau, etc. SIEM console

Security. Analytics. Insight.39 SIEM Integration - Advanced SIEM columnar (parquet) processing syslog data SQL (Impala, SparkSQL) H D F S index (ES) queue (Kafka) a c c e s s other data sources SIEM   connector raw logs SIEM console SQL and search   interface Tableau, Kibana, etc. requires parsing and formatting in a SIEM readable format (e.g., CEF)

Security. Analytics. Insight.40 What I am Working On Data Stores Analytics Forensics Models Admin 10.9.79.109 --> 3.16.204.150 10.8.24.80 --> 192.168.148.193 10.8.50.85 --> 192.168.148.193 10.8.48.128 --> 192.168.148.193 10.9.79.6 --> 192.168.148.193 10.9.79.6 10.8.48.128 80 53 8.8.8.8 127.0.0.1 Anomalies Decomposition Data Seasonal Trend Anomaly Details “Hunt” ExplainVisual Search • Big data backend • Own visualization engine (Web-based) • Visualization workﬂows

Security. Analytics. Insight.41 BlackHat Workshop Visual Analytics - Delivering Actionable Security Intelligence August 1-6 2015, Las Vegas, USA big data | analytics | visualization

Security. Analytics. Insight.42 http://secviz.org List: secviz.org/mailinglist Twitter: @secviz Share, discuss, challenge, and learn about security visualization. Security Visualization Community

Security. Analytics. Insight. raffael.marty@pixlcloud.com http://slideshare.net/zrlram http://secviz.org and @secviz Further resources: