Finding Needles in Haystacks          (the size of countries)                            Michael Baker                    ...
Acknowledgements                        David Turnbull @dsturnbull                        Gerald Kaszuba @gakman          ...
Two RulesSunday, 21 October 12
The Landscape.Sunday, 21 October 12
Exhibit A                        CVE-2011-3192 - “Apache Killer”                        auxiliary/dos/http/apache_range_do...
Prevention Fails.Sunday, 21 October 12
Detection is the key.Sunday, 21 October 12
NSM - “focused on providing an                intrusion analyst with the best possible                 information in the ...
Network Security Monitoring                        Advocates focus on detection and that                        prevention...
Tools collect.Sunday, 21 October 12
People analyze.Sunday, 21 October 12
Network Security Monitoring                        Squil                        Argus                        Flowgrep     ...
It’s all about Context.Sunday, 21 October 12
Context                        Enriched information, not just IP Addresses.                        Additional intelligence...
Full Packet Capture                        Complete record of all network data.                        Provides the highes...
NSM + FPC                        > % OPTIONSSunday, 21 October 12
bit.ly/RdrI6MSunday, 21 October 12
“The difficulty shifts from traffic                 collection to traffic analysis. If you can                  store hundreds o...
Big Data is a collection of data sets so large and                  complex that it becomes difficult to process            ...
Big Data                        Cloud - Elastic compute and Cheap Storage                        Map Reduce - parallel com...
Map ReduceSunday, 21 October 12
Sunday, 21 October 12
Big Data Scale                        I want to ask a 2.5TB question                          Process 2.5TB, 8 hours, 4 Co...
Big Data Scale                                    Complex Job (Approx 2.5TB)         500                        480       ...
History                        Google Map Reduce Whitepaper (2004)                        Google File System Whitepaper (2...
@packetpig                        @packetpig = Packets (FPC) + Pig                        Pig uses a data flow language cal...
Features                        Full access to IP packets at scale.                        Threat Analysis (Snort)        ...
Sunday, 21 October 12
Finding Zero DaysSunday, 21 October 12
Worth a coffee JD?                        Motivation                        Time window                        Attacker    ...
Attacker InformationSunday, 21 October 12
File ExtractionSunday, 21 October 12
Big Data                        Security AnalyticsSunday, 21 October 12
Anscombe’s Quartet                                  I                    II                    III                  IV    ...
Anscombe’s Quartet                        Source: http://visual.ly/anscombes-quartetSunday, 21 October 12
Big Data Security Analytics                        Visualization       Prediction and                                     ...
This is not SIEM.Sunday, 21 October 12
Not SIEM                        Full Fidelity                        Explore and explain the data (evidence).             ...
VisualisationSunday, 21 October 12
Full HD                        Play, Pause, RewindSunday, 21 October 12
Outlier DetectionSunday, 21 October 12
ClassificationSunday, 21 October 12
Sunday, 21 October 12
Novelty and OutliersSunday, 21 October 12
Sunday, 21 October 12
Entropy and Covert                             ChannelsSunday, 21 October 12
EnrichmentSunday, 21 October 12
GeocodingSunday, 21 October 12
TORSunday, 21 October 12
Torrent TriangulationSunday, 21 October 12
TransformationSunday, 21 October 12
Network Graphs and                           RelationshipsSunday, 21 October 12
Intelligence and Metric                    SharingSunday, 21 October 12
Indicators of Compromise                        OpenIOC and CyBOX                          Open Indicators of Compromise (...
DNS and MalwareSunday, 21 October 12
www.weatherzone.com.au                                            www.watoday.com.au                                      ...
Analytics or                        SurveillanceSunday, 21 October 12
bit.ly/TzcSq8Sunday, 21 October 12
Questions?                        @packetpig @packetloopSunday, 21 October 12
Thank you!                        http://blog.packetloop.comSunday, 21 October 12
Upcoming SlideShare
Loading in...5
×

Ruxcon Finding Needles in Haystacks (the size of countries)

917

Published on

This version of the presentation focused on finding zero days, some machine learning and security visualization.

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
917
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
17
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Ruxcon Finding Needles in Haystacks (the size of countries)

  1. 1. Finding Needles in Haystacks (the size of countries) Michael Baker @cloudjunky Ruxcon - October 2012Sunday, 21 October 12
  2. 2. Acknowledgements David Turnbull @dsturnbull Gerald Kaszuba @gakman Packetpig CommittersSunday, 21 October 12
  3. 3. Two RulesSunday, 21 October 12
  4. 4. The Landscape.Sunday, 21 October 12
  5. 5. Exhibit A CVE-2011-3192 - “Apache Killer” auxiliary/dos/http/apache_range_dos 2011-08-19 normal Apache Range header DoS (Apache Killer) Snort 1:19825 /Ranges*x3As*bytes=([dx2D]+x2C){50}/Hsmi /Ranges*x3As*bytes=([dx2D]+[x2Cs]*){50}/ HsmiSunday, 21 October 12
  6. 6. Prevention Fails.Sunday, 21 October 12
  7. 7. Detection is the key.Sunday, 21 October 12
  8. 8. NSM - “focused on providing an intrusion analyst with the best possible information in the shortest amount of time” - NSMWikiSunday, 21 October 12
  9. 9. Network Security Monitoring Advocates focus on detection and that prevention will fail. Believes in inventoried and defensible networks. Build entropy from alert (attack) information. Provide analysts with accurate information as fast as possible.Sunday, 21 October 12
  10. 10. Tools collect.Sunday, 21 October 12
  11. 11. People analyze.Sunday, 21 October 12
  12. 12. Network Security Monitoring Squil Argus Flowgrep Snort and Suricata Bro Network Miner NetwitnessSunday, 21 October 12
  13. 13. It’s all about Context.Sunday, 21 October 12
  14. 14. Context Enriched information, not just IP Addresses. Additional intelligence on attackers. Allow you to perform detective work What if? Branch analysis and exploring data. Providing full fidelity and full context quickly.Sunday, 21 October 12
  15. 15. Full Packet Capture Complete record of all network data. Provides the highest fidelity to analysts. Only way to really understand subtle, targeted attacks. Play, pause and rewind your network. No need to have a specific logging setup.Sunday, 21 October 12
  16. 16. NSM + FPC > % OPTIONSSunday, 21 October 12
  17. 17. bit.ly/RdrI6MSunday, 21 October 12
  18. 18. “The difficulty shifts from traffic collection to traffic analysis. If you can store hundreds of gigabytes of traffic per day, how do you make sense of it?” - Richard BejtlichSunday, 21 October 12
  19. 19. Big Data is a collection of data sets so large and complex that it becomes difficult to process using on-hand database management tools. The challenges include capture, curation, storage, search, sharing, analysis, and visualization. - WikipediaSunday, 21 October 12
  20. 20. Big Data Cloud - Elastic compute and Cheap Storage Map Reduce - parallel computation Pig, Hive - avoid writing M/R NoSQL - Cassandra and MongoSunday, 21 October 12
  21. 21. Map ReduceSunday, 21 October 12
  22. 22. Sunday, 21 October 12
  23. 23. Big Data Scale I want to ask a 2.5TB question Process 2.5TB, 8 hours, 4 Compute units. Process 2.5TB, 4 hours , 8 Compute units. Process 2.5TB, 2 hours, 16 Compute units. Process 2.5TB, 1 hour, 32 Compute units. Process 2.5 TB, 30 minutes, 64 Compute units. Process 2.5 TB , 15 minutes, 128 Compute units. Scale my compute to answer my question.Sunday, 21 October 12
  24. 24. Big Data Scale Complex Job (Approx 2.5TB) 500 480 Minutes 375 250 240 125 120 60 0 30 15 4 8 16 32 64 128Sunday, 21 October 12
  25. 25. History Google Map Reduce Whitepaper (2004) Google File System Whitepaper (2003) Hadoop is an Apache Project for M/R (2007) Hadoop File System is a distributed file system for Hadoop nodes (2007) Pig is a data analysis language to ease the creation of Map / Reduce jobs that run on Hadoop Clusters (2008)Sunday, 21 October 12
  26. 26. @packetpig @packetpig = Packets (FPC) + Pig Pig uses a data flow language called Pig Latin. Executes Map/Reduce Jobs over Hadoop Clusters. Works identically on-premise or in the cloud (Amazon’s EMR)Sunday, 21 October 12
  27. 27. Features Full access to IP packets at scale. Threat Analysis (Snort) Traffic Analysis. Flow-based deep packet inspection. Geo-Location Passive OS Detection (p0f) File DissectionSunday, 21 October 12
  28. 28. Sunday, 21 October 12
  29. 29. Finding Zero DaysSunday, 21 October 12
  30. 30. Worth a coffee JD? Motivation Time window Attacker Attack type Target Obfuscated AnonymisedSunday, 21 October 12
  31. 31. Attacker InformationSunday, 21 October 12
  32. 32. File ExtractionSunday, 21 October 12
  33. 33. Big Data Security AnalyticsSunday, 21 October 12
  34. 34. Anscombe’s Quartet I II III IV x y x y x y x y 0.0 8.04 10.0 9.14 10.0 7.46 8.0 6.58 8.0 6.95 8.0 8.14 8.0 6.77 8.0 5.76 13.0 7.58 13.0 8.74 13.0 12.74 8.0 7.71 9.0 8.81 9.0 8.77 9.0 7.11 8.0 8.84 11.0 8.33 11.0 9.26 11.0 7.81 8.0 8.47 14.0 9.96 14.0 8.10 14.0 8.84 8.0 7.04 6.0 7.24 6.0 6.13 6.0 6.08 8.0 5.25 4.0 4.26 4.0 3.10 4.0 5.39 19.0 12.50 12.0 10.84 12.0 9.13 12.0 8.15 8.0 5.56 7.0 4.82 7.0 7.26 7.0 6.42 8.0 7.91 5.0 5.68 5.0 4.74 5.0 5.73 8.0 6.89 Source: http://en.wikipedia.org/wiki/Anscombe%27s_quartetSunday, 21 October 12
  35. 35. Anscombe’s Quartet Source: http://visual.ly/anscombes-quartetSunday, 21 October 12
  36. 36. Big Data Security Analytics Visualization Prediction and Probability Fidelity Intelligence sharing Interaction Statistical Analysis Outlier Detection Feature Extraction Attacker Profiling Machine Learning Enrichment TransformSunday, 21 October 12
  37. 37. This is not SIEM.Sunday, 21 October 12
  38. 38. Not SIEM Full Fidelity Explore and explain the data (evidence). Play, Pause and Rewind. Blink and you miss it technology. No aggregation. No parsers or complex integration. Clear intelligence.Sunday, 21 October 12
  39. 39. VisualisationSunday, 21 October 12
  40. 40. Full HD Play, Pause, RewindSunday, 21 October 12
  41. 41. Outlier DetectionSunday, 21 October 12
  42. 42. ClassificationSunday, 21 October 12
  43. 43. Sunday, 21 October 12
  44. 44. Novelty and OutliersSunday, 21 October 12
  45. 45. Sunday, 21 October 12
  46. 46. Entropy and Covert ChannelsSunday, 21 October 12
  47. 47. EnrichmentSunday, 21 October 12
  48. 48. GeocodingSunday, 21 October 12
  49. 49. TORSunday, 21 October 12
  50. 50. Torrent TriangulationSunday, 21 October 12
  51. 51. TransformationSunday, 21 October 12
  52. 52. Network Graphs and RelationshipsSunday, 21 October 12
  53. 53. Intelligence and Metric SharingSunday, 21 October 12
  54. 54. Indicators of Compromise OpenIOC and CyBOX Open Indicators of Compromise (XML) Host and Network Indicators of Compromise Fork a github repository Execute Packetpig scripts that find bad things and visualise themSunday, 21 October 12
  55. 55. DNS and MalwareSunday, 21 October 12
  56. 56. www.weatherzone.com.au www.watoday.com.au www.tweednews.com.au www.triplem.com.au www.tradingroom.com.au www.tradingpost.com.au www.themorningbulletin.com.au www.theherald.com.au www.thechronicle.com.au www.theage.com.au www.sunshinecoastdaily.com.au www.stayz.com.au www.smhshop.com.au www.smartedition.smh.com.au www.rsvp.com.au www.qt.com.au www.portnews.com.au www.northerndailyleader.com.au www.magic1278.com.au www.investsmart.com.au www.goodguides.com.au www.fox.com.au dns$V1 www.fairfaxsyndication.com www.fairfaxevents.com.au www.facebook.com www.adcentre.com.au www.3aw.com.au twitter.com tributes.smh.com.au tradingroom.com.au subscriptions.fairfax.com.au smhshop.com.au magic1278.com.au m.smh.com.au investsmart.com.au goodguides.com.au fairfaxsyndication.com fairfaxevents.com.au dsa.f2.com.au.edgesuite.net classifieds.fairfax.com.au apndigital.com.au apm.com.au adcentre.com.au a1040.g.akamai.net 3aw.com.au 2000 4000 6000 8000 10000 dns$V2Sunday, 21 October 12
  57. 57. Analytics or SurveillanceSunday, 21 October 12
  58. 58. bit.ly/TzcSq8Sunday, 21 October 12
  59. 59. Questions? @packetpig @packetloopSunday, 21 October 12
  60. 60. Thank you! http://blog.packetloop.comSunday, 21 October 12
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×