Situational
Awareness

raffael marty - pixlcloud
     december 2011
Is this useful for Situational
                                Awareness?




pixlcloud |   creating big data stories                 copyright (c) 2011
Overview
 Network Security                          Sit Awareness   Today




  Where we should be                       Challenges      Resources




pixlcloud |   creating big data stories                            copyright © 2011
Raffael Marty

      • SaaS business expert                                              pixlcloud
      • Data visualization practitioner
      • Security data analyst
                                                                              IBM Research




                        Applied Security Visualization
                               Publisher: Addison Wesley (August, 2008)
                                           ISBN: 0321510100




pixlcloud |   creating big data stories                                                      copyright (c) 2011
Cyber Security
       Network Security                                       Information Security

       Data Collection                                        Authentication
                                                              Authorization
       Forensics / IR
                                                              Accounting
       Reporting                               Neglected!!!   BCM / DR
       Alerting                                               OS Security
       Situational Awareness                                  Policies and Procedures
                                                              ...

                            Reactive                                Pro-Active

pixlcloud |   creating big data stories                                                 copyright (c) 2011
Situational Awareness
 “Situational Awareness is the ability to identify, process, and
 comprehend the critical elements of information about what
 is happening to the team with regards to the mission. More
 simply, it’s knowing what is going on around you.”


                                           ‣ find air force viz images




                                                         IWViz - IDS Situational Awareness

pixlcloud |   creating big data stories                                         copyright © 2011
Sit Awareness Is Visualization
  ‣ Visualization - because machine centered approaches have failed

  ‣ Leverage human cognitive capabilities
     ‣Pattern recognition
     ‣Pre-attentive processing
     ‣Context memory




pixlcloud |   creating big data stories                         copyright © 2011
Today

pixlcloud |   creating big data stories           copyright (c) 2011
Data Sources for Sit Awareness
                                          1.1.1.1           10.0.0.2
        ‣Flow records
                                                    9.4.242.10



        ‣ Firewalls                       1.1.1.1            10.0.0.2

                                                    9.4.242.10


        ‣ IDS/IPSs                        1.1.1.1            10.0.0.2

                                                    9.4.242.10


     ‣ What about: PCAP, DNS, BGP, OS, Proxies, User behavior ??

     ‣ Context information - Hosts, Users, ...


pixlcloud |   creating big data stories                                 copyright © 2011
Todays Visualization Tools
  ‣ Based on specific data source
  ‣ Hard to use
  ‣ Limited interactivity
  ‣ Not real-time
  ‣ Slow
  ‣ Ugly


       ‣   Gephi                     ‣     PicViz
       ‣   R                         ‣     Treemap 4.1
       ‣   Matlab                    ‣     Google Earth
       ‣   Mondrian

pixlcloud |    creating big data stories                  copyright © 2011
Take the Blinders Off!




pixlcloud |   creating big data stories                     copyright © 2011
Visualization Maturity
      ‣ Data Collection                                   Contextual Data

                                                                                                                              iterations



      ‣ Data Analysis                      Data Sources     (Data Store)              Structured Data                   Visual Representation




      ‣ Context Integration                                                 parsing
                                                                                                        visualization


                                                                                                        feature selection



      ‣ Visualization
                                                            files
                                                            database
                                                                                       filtering
                                                                                       aggregation
                                                                                       cleansing


      ‣ Visual Analytics

      ‣ Collaboration

      ‣ Dissemination


pixlcloud |   creating big data stories                                                                                            copyright © 2011
Security Visualization Dichotomy
      Security                             Visualization
      ‣ security data                      ‣   types of data
      ‣ networking protocols               ‣   perception
      ‣ routing protocols (the Internet)   ‣   optics
      ‣ security impact                    ‣   color theory
      ‣ security policy                    ‣   depth cue theory
      ‣ jargon                             ‣   interaction theory
      ‣ use-cases                          ‣   types of graphs
      ‣ are the end-users                  ‣   human computer interaction
pixlcloud |   creating big data stories                                     copyright © 2011
Landscape Changes
   Threat Landscape                                     Technology
    • from       disruptive to disastrous                • Big   Data
    • from       audacious to “low and slow”              • NoSQL

    • from       fame to financial gain                    • Column-based    data stores
    • from       manual to automated                      • Map    Reduce (hadoop)
    • from       indiscriminate to targeted              • Cloud
    • from       infrastructure to applications           • on    demand computing

                                  We have technology to attack the threats!
                                   BUT we don’t know what to do with it!
pixlcloud |   creating big data stories                                                   copyright © 2011
The Public Sector
   ‣ Currently using a lot of Excel

   ‣ Big data technologies (e.g., Datameer, Karmasphere, Cloudera)

   ‣ Incremental improvements to SIEM tools (e.g., ArcSight, etc.)

   ‣ Using non security / network tools (e.g., Advizor, Cognos)



   ‣ Working with blacklists and whitelists

   ‣ Not understanding the data intrinsically



pixlcloud |   creating big data stories                              copyright © 2011
The Government
 Everything is different from Industry

              Scale                              Data sources
                   e.g., DISA has 5 million         e.g., ASIM CIDS
                         live hosts

               Types of attacks                   Adversaries
                    I have no example ....          e.g., Nation states


pixlcloud |   creating big data stories                                   copyright © 2011
We Need

pixlcloud |   creating big data stories             copyright (c) 2011
What we Need
   ‣ Leverage advanced technologies (big data, etc.)

   ‣ Build for the actual users, not programmers!

   ‣ End to end tools, not yet another library

   ‣ Interactive, not static!

   ‣ Multiple data sources at once

   ‣ Leverage context, not just event data

   ‣ Decouple data from the tools

   ‣ Crowd intelligence

pixlcloud |   creating big data stories                  copyright © 2011
Make it This Simple!




pixlcloud |   creating big data stories                    copyright © 2011
Challenges

pixlcloud |   creating big data stories            copyright (c) 2011
Maturity Challenge




              Companies and products are stuck on the left hand side!
pixlcloud |   creating big data stories                                 copyright © 2011
1
                                          Data Challenges
  ‣ No data - no insights - no sit awareness

  ‣ We don’t even have / collect the data

  ‣ It is too hard to collect data

  ‣ We don’t understand our data!

  ‣ Data silos

  ‣ Large amounts of semi-structured data
     ‣Parsing data is extremely hard



pixlcloud |   creating big data stories                     copyright © 2011
Tool Challenges
  ‣ Same old - all over                                           Overview first

     ‣Does your SIEM support visual analytics?

  ‣ Missing: Brushing, Interactivity

  ‣ Help the user understand the data!                            Zoom and Filter



  ‣ Highly scalable visualization systems are hard to build!

  ‣ What algorithms are useful? (e.g., clustering)
                                                                 Details on demand
  ‣ Visualization expertise is missing

  ‣ Visualization AND security is an interdisciplinary problem

pixlcloud |   creating big data stories                                 copyright © 2011
Visualization Challenges
   ‣ Skilled people are missing

   ‣ What are we even trying to look for?

   ‣ Anomaly detection is not working

   ‣ Academia is disconnected
      ‣Use-cases and problems
      ‣State of the art in industry

   ‣ Visualization is always an afterthought


pixlcloud |   creating big data stories              copyright © 2011
Myths
  ‣Real-time
     ‣Do we really need real-time?

  ‣Hadoop
     ‣Not everything that is big data needs to use Hadoop!

     ‣Know your technologies!

  ‣Cloud
     ‣Will we ever put security relevant data into the cloud?




pixlcloud |   creating big data stories                         copyright © 2011
Resources
  ‣ SecViz: http://secviz.org and @secviz

  ‣ CERT - NetSA: http://www.cert.org/netsa/
     ‣Mainly a collection of papers and links to some tools (SiLK)

  ‣ VizSec Conference: http://www.vizsec.org

  ‣ Applied Security Visualization
    R. Marty, 2008



pixlcloud |   creating big data stories                              copyright © 2011
pixlcloud                                              buy now



creating big data stories



                 @raffaelmarty
             copyright (c) by r. marty - december 2011

Security - Situational awareness

  • 1.
  • 2.
    Is this usefulfor Situational Awareness? pixlcloud | creating big data stories copyright (c) 2011
  • 3.
    Overview Network Security Sit Awareness Today Where we should be Challenges Resources pixlcloud | creating big data stories copyright © 2011
  • 4.
    Raffael Marty • SaaS business expert pixlcloud • Data visualization practitioner • Security data analyst IBM Research Applied Security Visualization Publisher: Addison Wesley (August, 2008) ISBN: 0321510100 pixlcloud | creating big data stories copyright (c) 2011
  • 5.
    Cyber Security Network Security Information Security Data Collection Authentication Authorization Forensics / IR Accounting Reporting Neglected!!! BCM / DR Alerting OS Security Situational Awareness Policies and Procedures ... Reactive Pro-Active pixlcloud | creating big data stories copyright (c) 2011
  • 6.
    Situational Awareness “SituationalAwareness is the ability to identify, process, and comprehend the critical elements of information about what is happening to the team with regards to the mission. More simply, it’s knowing what is going on around you.” ‣ find air force viz images IWViz - IDS Situational Awareness pixlcloud | creating big data stories copyright © 2011
  • 7.
    Sit Awareness IsVisualization ‣ Visualization - because machine centered approaches have failed ‣ Leverage human cognitive capabilities ‣Pattern recognition ‣Pre-attentive processing ‣Context memory pixlcloud | creating big data stories copyright © 2011
  • 8.
    Today pixlcloud | creating big data stories copyright (c) 2011
  • 9.
    Data Sources forSit Awareness 1.1.1.1 10.0.0.2 ‣Flow records 9.4.242.10 ‣ Firewalls 1.1.1.1 10.0.0.2 9.4.242.10 ‣ IDS/IPSs 1.1.1.1 10.0.0.2 9.4.242.10 ‣ What about: PCAP, DNS, BGP, OS, Proxies, User behavior ?? ‣ Context information - Hosts, Users, ... pixlcloud | creating big data stories copyright © 2011
  • 10.
    Todays Visualization Tools ‣ Based on specific data source ‣ Hard to use ‣ Limited interactivity ‣ Not real-time ‣ Slow ‣ Ugly ‣ Gephi ‣ PicViz ‣ R ‣ Treemap 4.1 ‣ Matlab ‣ Google Earth ‣ Mondrian pixlcloud | creating big data stories copyright © 2011
  • 11.
    Take the BlindersOff! pixlcloud | creating big data stories copyright © 2011
  • 12.
    Visualization Maturity ‣ Data Collection Contextual Data iterations ‣ Data Analysis Data Sources (Data Store) Structured Data Visual Representation ‣ Context Integration parsing visualization feature selection ‣ Visualization files database filtering aggregation cleansing ‣ Visual Analytics ‣ Collaboration ‣ Dissemination pixlcloud | creating big data stories copyright © 2011
  • 13.
    Security Visualization Dichotomy Security Visualization ‣ security data ‣ types of data ‣ networking protocols ‣ perception ‣ routing protocols (the Internet) ‣ optics ‣ security impact ‣ color theory ‣ security policy ‣ depth cue theory ‣ jargon ‣ interaction theory ‣ use-cases ‣ types of graphs ‣ are the end-users ‣ human computer interaction pixlcloud | creating big data stories copyright © 2011
  • 14.
    Landscape Changes Threat Landscape Technology • from disruptive to disastrous • Big Data • from audacious to “low and slow” • NoSQL • from fame to financial gain • Column-based data stores • from manual to automated • Map Reduce (hadoop) • from indiscriminate to targeted • Cloud • from infrastructure to applications • on demand computing We have technology to attack the threats! BUT we don’t know what to do with it! pixlcloud | creating big data stories copyright © 2011
  • 15.
    The Public Sector ‣ Currently using a lot of Excel ‣ Big data technologies (e.g., Datameer, Karmasphere, Cloudera) ‣ Incremental improvements to SIEM tools (e.g., ArcSight, etc.) ‣ Using non security / network tools (e.g., Advizor, Cognos) ‣ Working with blacklists and whitelists ‣ Not understanding the data intrinsically pixlcloud | creating big data stories copyright © 2011
  • 16.
    The Government Everythingis different from Industry Scale Data sources e.g., DISA has 5 million e.g., ASIM CIDS live hosts Types of attacks Adversaries I have no example .... e.g., Nation states pixlcloud | creating big data stories copyright © 2011
  • 17.
    We Need pixlcloud | creating big data stories copyright (c) 2011
  • 18.
    What we Need ‣ Leverage advanced technologies (big data, etc.) ‣ Build for the actual users, not programmers! ‣ End to end tools, not yet another library ‣ Interactive, not static! ‣ Multiple data sources at once ‣ Leverage context, not just event data ‣ Decouple data from the tools ‣ Crowd intelligence pixlcloud | creating big data stories copyright © 2011
  • 19.
    Make it ThisSimple! pixlcloud | creating big data stories copyright © 2011
  • 20.
    Challenges pixlcloud | creating big data stories copyright (c) 2011
  • 21.
    Maturity Challenge Companies and products are stuck on the left hand side! pixlcloud | creating big data stories copyright © 2011
  • 22.
    1 Data Challenges ‣ No data - no insights - no sit awareness ‣ We don’t even have / collect the data ‣ It is too hard to collect data ‣ We don’t understand our data! ‣ Data silos ‣ Large amounts of semi-structured data ‣Parsing data is extremely hard pixlcloud | creating big data stories copyright © 2011
  • 23.
    Tool Challenges ‣ Same old - all over Overview first ‣Does your SIEM support visual analytics? ‣ Missing: Brushing, Interactivity ‣ Help the user understand the data! Zoom and Filter ‣ Highly scalable visualization systems are hard to build! ‣ What algorithms are useful? (e.g., clustering) Details on demand ‣ Visualization expertise is missing ‣ Visualization AND security is an interdisciplinary problem pixlcloud | creating big data stories copyright © 2011
  • 24.
    Visualization Challenges ‣ Skilled people are missing ‣ What are we even trying to look for? ‣ Anomaly detection is not working ‣ Academia is disconnected ‣Use-cases and problems ‣State of the art in industry ‣ Visualization is always an afterthought pixlcloud | creating big data stories copyright © 2011
  • 25.
    Myths ‣Real-time ‣Do we really need real-time? ‣Hadoop ‣Not everything that is big data needs to use Hadoop! ‣Know your technologies! ‣Cloud ‣Will we ever put security relevant data into the cloud? pixlcloud | creating big data stories copyright © 2011
  • 26.
    Resources ‣SecViz: http://secviz.org and @secviz ‣ CERT - NetSA: http://www.cert.org/netsa/ ‣Mainly a collection of papers and links to some tools (SiLK) ‣ VizSec Conference: http://www.vizsec.org ‣ Applied Security Visualization R. Marty, 2008 pixlcloud | creating big data stories copyright © 2011
  • 27.
    pixlcloud buy now creating big data stories @raffaelmarty copyright (c) by r. marty - december 2011