Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
"Everest for me, and I believe for the world, is the physical and                         symbolic manifestation of overco...
About me• Employed by Six3 Systems, based in Fulton MD• Defensive Analytic Developer (CND-OPS)• Decade of solutions in Ana...
What is SHERPASURFINGOpen source Cyber Security Solution, providing aframework, base set of proven services, data sources,...
Topics• The Problem• Is there a better way?• Conclusions• QuestionsHadoop World 2011          cloudera Six3 Systems
The ProblemHadoop World 2011   cloudera Six3 Systems
Forces and Economics driving Cyber Security        Hackers…        Potential Victims…        Defenders…Hadoop World 2011  ...
A Story: Aftershock Widget Corporation Aftershock Widget Corporation                                                      ...
Aftershock Widget Corporation calls for HELP!We are going to bring in some smart people   Who brought in more smart people...
Is there a better way ?Hadoop World 2011         cloudera Six3 Systems
Driving tenants of SHERPA:• Cost effective scaling for handling BIGDATA• Brings all data together• Must support all forms ...
Assess the environment:• What are my data sources, how much data, how fast?• What are the data formats?• What do I need to...
Enough Talk ! Let’s get Started                                                                    TODO List              ...
Sources, Sinks and Agents       Users                  Intellectual             Protected                                 ...
The Pieces come Together                              SHERPA – Analytic Framework                                         ...
Develop and Deploy Analytics     Risk Potential                   Correlate           Perform         Index               ...
SHERPASURFING Toolkit • FLUME Sinks, Decorators • HBASE Object Definitions • Multiple forms of Enrichment • SHERPA Develop...
ConclusionsHadoop World 2011   cloudera Six3 Systems
The Wrap-up • The threat is very real, well funded and determined • The problem has an incredible often hidden impact • Ap...
sherpasurfing@gmail.com  QUESTIONS?“Imagination is more important than knowledge. For knowledge is limited to all wenow kn...
Upcoming SlideShare
Loading in …5
×

Hadoop World 2011: Sherpasurfing - Wayne Wheeles

4,472 views

Published on

Consider this - each day, billions of packets both benign and some malicious flow in and out of networks. The ability to survive the sheer volume of data, bring the NETFLOW data to rest, enrich it, correlate it and perform analysis is essential tasks of the modern Defensive Cyber Security Organization. SHERPASURFING is an open source platform built on the proven Cloudera stack enabling organizations to perform the Cyber Security mission at scale at an affordable price point. This session will include an overview of the solution, presentation of components and a demonstration of analytics.

Published in: Technology
  • Be the first to comment

Hadoop World 2011: Sherpasurfing - Wayne Wheeles

  1. 1. "Everest for me, and I believe for the world, is the physical and symbolic manifestation of overcoming odds to achieve a dream." ~Tom WhittakerSherpasurfingOpen Source Cyber Security SolutionWayne Wheeles, Six3 SystemsActive Defensive Analytic DeveloperHadoop World 2011 cloudera Six3 Systems
  2. 2. About me• Employed by Six3 Systems, based in Fulton MD• Defensive Analytic Developer (CND-OPS)• Decade of solutions in Analytics & Big data• 18 analytics in production, 12 forms of enrichmentHadoop World 2011 cloudera Six3 Systems
  3. 3. What is SHERPASURFINGOpen source Cyber Security Solution, providing aframework, base set of proven services, data sources,how to guides and patterns for analytic developmentbuilt on top of the Apache Hadoop stackHadoop World 2011 cloudera Six3 Systems
  4. 4. Topics• The Problem• Is there a better way?• Conclusions• QuestionsHadoop World 2011 cloudera Six3 Systems
  5. 5. The ProblemHadoop World 2011 cloudera Six3 Systems
  6. 6. Forces and Economics driving Cyber Security Hackers… Potential Victims… Defenders…Hadoop World 2011 cloudera Six3 Systems
  7. 7. A Story: Aftershock Widget Corporation Aftershock Widget Corporation PROFILE • Software development firm that develops applications for variety of different platforms A series of bizarre fraudulent charges appeared on 2009 – Credit card theft victimized 11.1M Aftershock Widgets credit cards Americans costing 54B The next generation application that Aftershock has 2009 – Intellectual designed is stolen and hits market six months Property cost the economy 1.2T dollars annually before release by a competitor In a recent poll 94% of During peak ordering season web-traffic grinds to a respondents stated that halt a DDOS was a major concernHadoop World 2011 cloudera Six3 Systems
  8. 8. Aftershock Widget Corporation calls for HELP!We are going to bring in some smart people Who brought in more smart people Who brought even more smart people ! Who brought in some off the shelf technology solution, resulting in a MARKETECTURE clear and compelling roadmap for the organization Their solution was butts in seats, big iron and huge integration costs Hadoop World 2011 cloudera Six3 Systems
  9. 9. Is there a better way ?Hadoop World 2011 cloudera Six3 Systems
  10. 10. Driving tenants of SHERPA:• Cost effective scaling for handling BIGDATA• Brings all data together• Must support all forms of data• Must be question agnostic• Foster sharing and exchanging of analytics/tradecraftHadoop World 2011 cloudera Six3 Systems
  11. 11. Assess the environment:• What are my data sources, how much data, how fast?• What are the data formats?• What do I need to know from the collected data?• What do I do with the information?• Who needs the results and what do we do with results ?Hadoop World 2011 cloudera Six3 Systems
  12. 12. Enough Talk ! Let’s get Started TODO List 1.) Need some commodity X 5 Hardware, configure network32GB RAM4x300GB 6G SAS 15k HDD8-Core AMD Opteron Processor Model 6128 (2.0GHz, 80W) 2.) Provision them with RHEL X86_64 Server 6.1 3.) Install JDK 1.6u26 4.) Install the Cloudera CDH3U1, Enterprise 3.5.2 5.) Configure HDFS, HBASE,HDFS - 4 Data Nodes/Task Trackers, Name Node ZOOKEEPER, FLUMEHBASE – 1 Master, 4 Region ServersZookeeper – 4 Zookeeper serversHUE - 1 HUE, 4 HUE AgentsFLUME – 1 Master, 2 NodesHadoop World 2011 cloudera Six3 Systems
  13. 13. Sources, Sinks and Agents Users Intellectual Protected TODO List Property Data 1.) Identify data sources of potential value 2.) Install FLUME onUser Logging each source from Cloudera Intrusion Corporate Detection Enterprise App Server(s) System(s) 3.) Configure FLUME Agent to tailsink IPS/IDS signature hits a file or directory for each sourceLog data 4.) Test each data source to ensure data is being collected correctly sink using command line (dump) Corporate Firewall Firewall logs $ flume dump text("/cp/10/21/0800/current.log") sink sink 5.) Configure the sink(destination) Flow Data Flow Capture for each FLUME Agent Packet Capture Aftershock Corporate Internet Gateway 1Hadoop World 2011 sink cloudera Six3 Systems
  14. 14. The Pieces come Together SHERPA – Analytic Framework SHERPA Components HUE HBASE PIG HIVE Data Sets GEO Enrichment S 30 days 61,764,205 netflows T 30 days 1,065,977 SNORT SQOOP Port Enrichment A T 30 days 4,065,977 Firewall Protocol Enrichment S Packet Data HDFS Firewall Logs ZOOKEEPER Netflow Data Application Server Logging User Logging IDS/IPS Logs sink sink sink sink sink sink User App Netflow Packet Firewall IDS/IPS Logging Server Logging Capture Logs Logs Hadoop World 2011 Logging cloudera Six3 Systems
  15. 15. Develop and Deploy Analytics Risk Potential Correlate Perform Index Report Analytic Results Enrichment Flow Characterization Analytic Analytic Runtime Environment Health & Analytic CORE Data Services Job Control SHERPA Status Registry Services SDK SHERPA – Analytic FrameworkHadoop World 2011 cloudera Six3 Systems
  16. 16. SHERPASURFING Toolkit • FLUME Sinks, Decorators • HBASE Object Definitions • Multiple forms of Enrichment • SHERPA Developers Guide and Cookbook • Two Sample Analytics • Enterprise Analytic FrameworkHadoop World 2011 cloudera Six3 Systems
  17. 17. ConclusionsHadoop World 2011 cloudera Six3 Systems
  18. 18. The Wrap-up • The threat is very real, well funded and determined • The problem has an incredible often hidden impact • Apache Hadoop stack provide an effective foundation • SHERPA solution builds on that stack • Provides a framework for Cyber Security AnalyticsHadoop World 2011 cloudera Six3 Systems
  19. 19. sherpasurfing@gmail.com QUESTIONS?“Imagination is more important than knowledge. For knowledge is limited to all wenow know and understand, while imagination embraces the entire world, and allthere ever will be to know and understand.”~Albert EinsteinHadoop World 2011 cloudera Six3 Systems

×