Your SlideShare is downloading. ×
0
Cloud-based Log Analysis and Visualization
                        RMLL 2010, Bordeaux, France
                           ...
Raffael (Raffy) Marty
       • Founder @
       • Chief Security Strategist and Product Manager @ Splunk
       • Manager ...
Agenda
            •Introduction                            •Do it Yourself

            •Visualization                   ...
Open Your Eyes




                        Logging as a Service   4         (c) by Raffael Marty
Tuesday, July 6, 2010
Security Is About Seeing




                        Logging as a Service   5   (c) by Raffael Marty
Tuesday, July 6, 2010
Goals
       - Learn how you can
          - use visualization to help solve security problems
          - leverage the cl...
Information Visualization?

                           A picture is worth a thousand log records.


                      ...
Visualization
                        and The Cloud
                                   8




Tuesday, July 6, 2010
InfoViz Process




        Collect                                Process             Visualize
        •large-scale data...
Collect
                                  10




Tuesday, July 6, 2010
Log Management
         • Log Collection and Centralization
         • Log Storage
         • Log Filtering
         • Log...
Process
                                  12




Tuesday, July 6, 2010
Standard Formats
          • Multiple formats
              Oct 13 20:00:43.874401 rule 193/0(match): block in on xl0: 212...
Normalization
          • Parsers
                        “To analyze or separate (input, for example) into more easily
  ...
Visualize
                                15




Tuesday, July 6, 2010
Choose Your Poison




                        Logging as a Service   16     (c) by Raffael Marty
Tuesday, July 6, 2010
Reporting vs. Visualization
          • Reporting Libraries                     • Visualization Libraries
               -...
HighCharts



    • Click-Through
    • On load
        - near real-time updates                   • AJAX data input via J...
Google Visualization API


           http://code.google.com/apis/visualization/interactive_charts.html

           • Java...
ProtoVis
          • JavaScript based visualization library
          • Charting
          • Treemaps
          • BoxPlots...
TheJIT   http://thejit.org/

          • JavaScript InfoVis Toolkit
          • Interactive
          • Link Graphs




  ...
Processing
          •Visualization library
          •Java based
          •Interactive (event handling)
          •Numbe...
Building Your Own

                                    23




Tuesday, July 6, 2010
Build Your Own




                                                          AfterGlow
                Loggly             ...
Data Collection in
                        the Cloud
                                    25




Tuesday, July 6, 2010
The (public) Cloud
         What it is                            Types
          • multi-tenancy                      • S...
LaaS - Logging as a Service
       • All your data in one place
          • Loggly manages your data (index, store, archiv...
Loggly Architecture
                                                                                Loggly
        Data So...
Loggly APIs
       • URL format:                                     http://wiki.loggly.com/api-documentation

           ...
Search
               http://[domain].loggly.com/api/search?q=404
               {
                    "data": [
         ...
Parser
                              Oct 13 20:00:38.018152 rule 57/0(match): pass in on xl1: 195.141.69.45.1030 > 62.2.32...
Visualize
                                  Parser              AfterGlow              Grapher

                          ...
AfterGlow Cloud
                                               Grapher   Loggly


                                        ...
Google Vis
          • JSON to Graphs
          • DataTable
               - used among all charts

          • Interactiv...
<script type="text/javascript">
                                           Google Vis Code
           google.load('visuali...
Visualization Use-Cases

                                      36




Tuesday, July 6, 2010
NetFlow Visualization
          • Treemap
          • Protovis.JS
          • Size: Amount
          • Brightness: Varianc...
Firewall Treemap




                        Logging as a Service   38        (c) by Raffael Marty
Tuesday, July 6, 2010
Firewall Log
                              Port                Source IP   Destination IP




                        Logg...
Visualization Resources


                                      40




Tuesday, July 6, 2010
http://secviz.org
                          Share, discuss, challenge, and learn about security
                          ...
Applied Security Visualization
        • Bridging the gap between security and visualization
        • Hands-on, end to en...
Thank You!




                        raffael.marty@loggly.com
                                 @zrlram


               ...
Upcoming SlideShare
Loading in...5
×

Cloud Log Analysis and Visualization

8,647

Published on

Cloud computing has changed the way businesses operate, the way businesses make money, and the way business have to protect their assets and information. More and more software applications are moving into the cloud. People are running their proxies in the cloud and soon you will be collecting your logs in the cloud. You shouldn't have to deal with log collection and log management. You should be able to focus your time on getting value out of the logs; to do log analysis and visualization.

In this presentation we will explore how we can leverage the cloud to build security visualization tools. We will discuss some common visualization libraries and have a look at how they can be deployed to solve security problems. We will see how easy it is to quickly stand up such an application. To close the presentation, we will look at a number of security visualization examples that show how security data benefits from visual representations. For example, how can network traffic, firewall data, or IDS data be visualized effectively?

Published in: Technology, Education
1 Comment
19 Likes
Statistics
Notes
  • Try logging to the cloud... http://loggr.net
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
No Downloads
Views
Total Views
8,647
On Slideshare
0
From Embeds
0
Number of Embeds
5
Actions
Shares
0
Downloads
329
Comments
1
Likes
19
Embeds 0
No embeds

No notes for slide

Transcript of "Cloud Log Analysis and Visualization"

  1. 1. Cloud-based Log Analysis and Visualization RMLL 2010, Bordeaux, France mobile-166 My syslog Raffael Marty - @zrlram Tuesday, July 6, 2010
  2. 2. Raffael (Raffy) Marty • Founder @ • Chief Security Strategist and Product Manager @ Splunk • Manager Solutions @ ArcSight • Intrusion Detection Research @ IBM Research • IT Security Consultant @ PriceWaterhouse Coopers Applied Security Visualization Publisher: Addison Wesley (August, 2008) ISBN: 0321510100 Logging as a Service 2 (c) by Raffael Marty Tuesday, July 6, 2010
  3. 3. Agenda •Introduction •Do it Yourself •Visualization •AfterGlow •Google Visualization API •InfoViz Process •Visualization Use-Cases •Visualization Tools •Visualization Resources •The Cloud •Loggly Logging as a Service 3 (c) by Raffael Marty Tuesday, July 6, 2010
  4. 4. Open Your Eyes Logging as a Service 4 (c) by Raffael Marty Tuesday, July 6, 2010
  5. 5. Security Is About Seeing Logging as a Service 5 (c) by Raffael Marty Tuesday, July 6, 2010
  6. 6. Goals - Learn how you can - use visualization to help solve security problems - leverage the cloud to build security visualization tools Logging as a Service 6 (c) by Raffael Marty Tuesday, July 6, 2010
  7. 7. Information Visualization? A picture is worth a thousand log records. Inspire Explore and Discover Answer a Pose a New Increase Communicate Support Question Question Efficiency Information Decisions Logging as a Service 7 (c) by Raffael Marty Tuesday, July 6, 2010
  8. 8. Visualization and The Cloud 8 Tuesday, July 6, 2010
  9. 9. InfoViz Process Collect Process Visualize •large-scale data collection •Your parsers •Visualization Tools •and processing •Standard formats •and Libraries Logging as a Service 9 (c) by Raffael Marty Tuesday, July 6, 2010
  10. 10. Collect 10 Tuesday, July 6, 2010
  11. 11. Log Management • Log Collection and Centralization • Log Storage • Log Filtering • Log Aggregation • Log Search and Extraction • Log Retention and Archiving Logging as a Service 11 (c) by Raffael Marty Tuesday, July 6, 2010
  12. 12. Process 12 Tuesday, July 6, 2010
  13. 13. Standard Formats • Multiple formats Oct 13 20:00:43.874401 rule 193/0(match): block in on xl0: 212.251.89.126.3859 >: S 1818630320:1818630320(0) win 65535 <mss 1460,nop,nop,sackOK> (DF) Oct 13 20:00:43 fwbox local4:warn|warning fw07 %PIX-4-106023: Deny tcp src internet: 212.251.89.126/3859 dst 212.254.110.98/135 by access-group "internet_access_in" Oct 13 20:00:43 fwbox kernel: DROPPED IN=eth0 OUT= MAC=ff:ff:ff:ff:ff:ff:00:0f:cc: 81:40:94:08:00 SRC=212.251.89.126 DST=212.254.110.98 LEN=576 TOS=0x00 PREC=0x00 TTL=255 ID=8624 PROTO=TCP SPT=3859 DPT=135 LEN=556 • Log Standards ‣ CEE (cee.mitre.org) ‣ SDEE ‣ WELF ‣ IDMEF ‣ CBE ‣ XDAS Logging as a Service 13 (c) by Raffael Marty Tuesday, July 6, 2010
  14. 14. Normalization • Parsers “To analyze or separate (input, for example) into more easily processed components.” (answers.com) • Generate a common output format for vis-tools (e.g., CSV) • For example ‣ Regex /(d{1,3}.d{1,3}.d{1,3}.d{1,3})/g ‣ http://secviz.org/content/parser-exchange Logging as a Service 14 (c) by Raffael Marty Tuesday, July 6, 2010
  15. 15. Visualize 15 Tuesday, July 6, 2010
  16. 16. Choose Your Poison Logging as a Service 16 (c) by Raffael Marty Tuesday, July 6, 2010
  17. 17. Reporting vs. Visualization • Reporting Libraries • Visualization Libraries - HighCharts - TheJIT - Flot - Graphael - Google Chart API - Protovis - Open Flash Chart - ProcessingJS - Flare JavaScript vs. Flash vs. XYZ Logging as a Service 17 (c) by Raffael Marty Tuesday, July 6, 2010
  18. 18. HighCharts • Click-Through • On load - near real-time updates • AJAX data input via JSON • Zoom http://www.highcharts.com/ Logging as a Service 18 (c) by Raffael Marty Tuesday, July 6, 2010
  19. 19. Google Visualization API http://code.google.com/apis/visualization/interactive_charts.html • JavaScript • Based on DataTables() • Many graphs • Playground - http://code.google.com/apis/ajax/playground Logging as a Service 19 (c) by Raffael Marty Tuesday, July 6, 2010
  20. 20. ProtoVis • JavaScript based visualization library • Charting • Treemaps • BoxPlots • Parallel Coordinates • etc. http://vis.stanford.edu/protovis/ Logging as a Service 20 (c) by Raffael Marty Tuesday, July 6, 2010
  21. 21. TheJIT http://thejit.org/ • JavaScript InfoVis Toolkit • Interactive • Link Graphs Logging as a Service 21 (c) by Raffael Marty Tuesday, July 6, 2010
  22. 22. Processing •Visualization library •Java based •Interactive (event handling) •Number of libraries to -draw in OpenGL -read XML files -write PDF files •Processing JS -JavaScript -HTML 5 Canvas http://processingjs.org/ -Web IDE http://processing.org/ Logging as a Service 22 (c) by Raffael Marty Tuesday, July 6, 2010
  23. 23. Building Your Own 23 Tuesday, July 6, 2010
  24. 24. Build Your Own AfterGlow Loggly Regexes Google Vis Logging as a Service 24 (c) by Raffael Marty Tuesday, July 6, 2010
  25. 25. Data Collection in the Cloud 25 Tuesday, July 6, 2010
  26. 26. The (public) Cloud What it is Types • multi-tenancy • SaaS - Software • elastic • PaaS - Platform • “infinite” resources • IaaS - Infrastructure • pay as you go Benefits • self provisioning • No installation • No elaborate configurations It’s not • No maintenance • private data center • Great scalability • virtualization • 7x24 availability Logging as a Service 26 (c) by Raffael Marty Tuesday, July 6, 2010
  27. 27. LaaS - Logging as a Service • All your data in one place • Loggly manages your data (index, store, archive, etc.) • Extremely fast search across all your data • Data source agnostic (no parsers) • Data management • access control • data segregation • data overview and summaries • API access Logging as a Service 27 (c) by Raffael Marty Tuesday, July 6, 2010
  28. 28. Loggly Architecture Loggly Data Sources Clients user interface mobile-166 My syslog Data collection API Data access Proxies Distributed Indexers and Search Machines indexing and processing Distributed data store Logging as a Service 28 (c) by Raffael Marty Tuesday, July 6, 2010
  29. 29. Loggly APIs • URL format: http://wiki.loggly.com/api-documentation http://<subdomain>.loggly.com/api/<resource> • RESTful API HTTP Based - Access through: /api/<resource> •GET - read - JSON, XML, JSONP output •POST - create • Authentication •PUT - update - Basic auth •DELETE - delete - oAuth http://loggly.loggly.com/api/search/?q=error syslog to: User: guest / Password: loggly logs.loggly.com:514 Logging as a Service 29 (c) by Raffael Marty Tuesday, July 6, 2010
  30. 30. Search http://[domain].loggly.com/api/search?q=404 { "data": [ { "indexed": "2010-07-03T17:17:38.909Z", "ip": "75.101.249.172", "text": "Oct 13 20:00:38.018152 rule 57/0(match): pass in on xl1: 195.141.69.45.1030 > 62.2.32.250.53: 34388 [1au] [|domain] (DF)", "inputname": "logglyweb", "timestamp": "2010-07-03 10:17:38" }, { "indexed": "2010-07-03T17:17:37.879Z", "ip": "75.101.249.172", "text": "Oct 13 20:00:38.115862 rule 57/0(match): pass in on xl1: 195.141.69.45.1030 > 192.134.0.49.53: 49962 [1au] [|domain] (DF)", "inputname": "logglyapp", "timestamp": "2010-07-03 10:17:37" }, ... Logging as a Service 30 (c) by Raffael Marty Tuesday, July 6, 2010
  31. 31. Parser Oct 13 20:00:38.018152 rule 57/0(match): pass in on xl1: 195.141.69.45.1030 > 62.2.32.250.53: 34388 [1au][|domain] (DF) Raw Oct 13 20:00:38.115862 rule 57/0(match): pass in on xl1: 195.141.69.45.1030 > 192.134.0.49.53: 49962 [1au][|domain] (DF) Oct 13 20:00:38.157238 rule 57/0(match): pass in on xl1: 195.141.69.45.1030 > 194.25.2.133.53: 14434 [1au][|domain] (DF) (.*) rule ([-d]+/d+)(.*?): (pass|block) (in|out) on (w+): (d+.d+.d+.d+).?(d*) [<>] Regex / Parser (d+.d+.d+.d+).?(d*): (.*) Oct 13 20:00:38.018152,57/0,match,pass,in,xl1,195.141.69.45,1030,62.2.32.250,53,34388 [1au][|domain] (DF) Normalized Oct 13 20:00:38.115862,57/0,match,pass,in,xl1,195.141.69.45,1030,192.134.0.49,53,49962 [1au][|domain] (DF) (CSV) Oct 13 20:00:38.157238,57/0,match,pass,in,xl1,195.141.69.45,1030,194.25.2.133,53,14434 [1au][|domain] (DF) Logging as a Service 31 (c) by Raffael Marty Tuesday, July 6, 2010
  32. 32. Visualize Parser AfterGlow Grapher CSV file Graph file digraph structs { graph [label="AfterGlow 1.5.8", fontsize=8]; node [shape=ellipse, style=filled, Configuration fontsize=10, width=1, height=1, fixedsize=true]; edge [len=1.6]; color.source=“green” if ($fields[0] ne “d”) "aaelenes" -> "Printing Resume" ; cluster.target=regex_replace("(d+).")."/8" "abbe" -> "Information Encryption" ; threshold.event=5 "aanna" -> "Patent Access" ; size.target=$fields[1] "aatharuv" -> "Ping" ; } http://afterglow.sf.net Logging as a Service 32 (c) by Raffael Marty Tuesday, July 6, 2010
  33. 33. AfterGlow Cloud Grapher Loggly JSON CSV DOT Graph Logging as a Service 33 (c) by Raffael Marty Tuesday, July 6, 2010
  34. 34. Google Vis • JSON to Graphs • DataTable - used among all charts • Interactivity through events Logging as a Service 34 (c) by Raffael Marty Tuesday, July 6, 2010
  35. 35. <script type="text/javascript"> Google Vis Code google.load('visualization', '1', {'packages':['motionchart', 'table', 'annotatedtimeline']}); google.setOnLoadCallback(call); var trends = new Array(); function call() { l! a $.ajax({ url: "http://logdog.loggly.com/api/search/?q=404&facets=True&buckets=100", n type:'GET', dataType: 'jsonp', username: 'xxxxx', password: 'xxxxxx', io success: function(data) { trends = data.data drawChart(); c t n } u }); f } t function drawChart() { o var data = new google.visualization.DataTable(); n data.addColumn('string', 'Search'); data.addColumn('datetime', 'Date'); is data.addColumn('number', 'Count'); e data.addRows(trends); od var chart = new google.visualization.MotionChart(document.getElementById('chart_div')); c chart.draw(data, {width: 600, height:300, state:state}); is var view = new google.visualization.DataView(data); h view.setRows(view.getFilteredRows([{column: 1, minValue: new Date(2007, 0, 1)}])); T var table = new google.visualization.Table(document.getElementById('test_dataview')); table.draw(view, {sortColumn: 1}); var time = new google.visualization.AnnotatedTimeLine(document.getElementById('timeline')); time.draw(timedata, {displayAnnotations: true}); } </script> Logging as a Service 35 (c) by Raffael Marty Tuesday, July 6, 2010
  36. 36. Visualization Use-Cases 36 Tuesday, July 6, 2010
  37. 37. NetFlow Visualization • Treemap • Protovis.JS • Size: Amount • Brightness: Variance • Color: Sensor • Shows: Scans - bright spots • Thanks to Chris Horsley Logging as a Service 37 (c) by Raffael Marty Tuesday, July 6, 2010
  38. 38. Firewall Treemap Logging as a Service 38 (c) by Raffael Marty Tuesday, July 6, 2010
  39. 39. Firewall Log Port Source IP Destination IP Logging as a Service 39 (c) by Raffael Marty Tuesday, July 6, 2010
  40. 40. Visualization Resources 40 Tuesday, July 6, 2010
  41. 41. http://secviz.org Share, discuss, challenge, and learn about security visualization. • List: secviz.org/mailinglist • Twitter: @secviz Logging as a Service 41 (c) by Raffael Marty Tuesday, July 6, 2010
  42. 42. Applied Security Visualization • Bridging the gap between security and visualization • Hands-on, end to end examples • Data processing and analysis Chapters • Visualization • Compliance • Data Sources • Insider Threat • From Data to Graphs • Visualization Tools Addison Wesley (August, 2008) • Perimeter Threat ISBN: 0321510100 Logging as a Service 42 (c) by Raffael Marty Tuesday, July 6, 2010
  43. 43. Thank You! raffael.marty@loggly.com @zrlram 43 Tuesday, July 6, 2010
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×