Hadoop World 2011: Sherpasurfing - Wayne Wheeles

"Everest for me, and I believe for the world, is the physical and symbolic manifestation of overcoming odds to achieve a dream." ~Tom WhittakerSherpasurfingOpen Source Cyber Security SolutionWayne Wheeles, Six3 SystemsActive Defensive Analytic DeveloperHadoop World 2011 cloudera Six3 Systems

About me• Employed by Six3 Systems, based in Fulton MD• Defensive Analytic Developer (CND-OPS)• Decade of solutions in Analytics & Big data• 18 analytics in production, 12 forms of enrichmentHadoop World 2011 cloudera Six3 Systems

What is SHERPASURFINGOpen source Cyber Security Solution, providing aframework, base set of proven services, data sources,how to guides and patterns for analytic developmentbuilt on top of the Apache Hadoop stackHadoop World 2011 cloudera Six3 Systems

Topics• The Problem• Is there a better way?• Conclusions• QuestionsHadoop World 2011 cloudera Six3 Systems

The ProblemHadoop World 2011 cloudera Six3 Systems

Forces and Economics driving Cyber Security Hackers… Potential Victims… Defenders…Hadoop World 2011 cloudera Six3 Systems

A Story: Aftershock Widget Corporation Aftershock Widget Corporation PROFILE • Software development firm that develops applications for variety of different platforms A series of bizarre fraudulent charges appeared on 2009 – Credit card theft victimized 11.1M Aftershock Widgets credit cards Americans costing 54B The next generation application that Aftershock has 2009 – Intellectual designed is stolen and hits market six months Property cost the economy 1.2T dollars annually before release by a competitor In a recent poll 94% of During peak ordering season web-traffic grinds to a respondents stated that halt a DDOS was a major concernHadoop World 2011 cloudera Six3 Systems

Aftershock Widget Corporation calls for HELP!We are going to bring in some smart people Who brought in more smart people Who brought even more smart people ! Who brought in some off the shelf technology solution, resulting in a MARKETECTURE clear and compelling roadmap for the organization Their solution was butts in seats, big iron and huge integration costs Hadoop World 2011 cloudera Six3 Systems

Is there a better way ?Hadoop World 2011 cloudera Six3 Systems

Driving tenants of SHERPA:• Cost effective scaling for handling BIGDATA• Brings all data together• Must support all forms of data• Must be question agnostic• Foster sharing and exchanging of analytics/tradecraftHadoop World 2011 cloudera Six3 Systems

Assess the environment:• What are my data sources, how much data, how fast?• What are the data formats?• What do I need to know from the collected data?• What do I do with the information?• Who needs the results and what do we do with results ?Hadoop World 2011 cloudera Six3 Systems

Enough Talk ! Let’s get Started TODO List 1.) Need some commodity X 5 Hardware, configure network32GB RAM4x300GB 6G SAS 15k HDD8-Core AMD Opteron Processor Model 6128 (2.0GHz, 80W) 2.) Provision them with RHEL X86_64 Server 6.1 3.) Install JDK 1.6u26 4.) Install the Cloudera CDH3U1, Enterprise 3.5.2 5.) Configure HDFS, HBASE,HDFS - 4 Data Nodes/Task Trackers, Name Node ZOOKEEPER, FLUMEHBASE – 1 Master, 4 Region ServersZookeeper – 4 Zookeeper serversHUE - 1 HUE, 4 HUE AgentsFLUME – 1 Master, 2 NodesHadoop World 2011 cloudera Six3 Systems

Sources, Sinks and Agents Users Intellectual Protected TODO List Property Data 1.) Identify data sources of potential value 2.) Install FLUME onUser Logging each source from Cloudera Intrusion Corporate Detection Enterprise App Server(s) System(s) 3.) Configure FLUME Agent to tailsink IPS/IDS signature hits a file or directory for each sourceLog data 4.) Test each data source to ensure data is being collected correctly sink using command line (dump) Corporate Firewall Firewall logs $ flume dump text("/cp/10/21/0800/current.log") sink sink 5.) Configure the sink(destination) Flow Data Flow Capture for each FLUME Agent Packet Capture Aftershock Corporate Internet Gateway 1Hadoop World 2011 sink cloudera Six3 Systems

The Pieces come Together SHERPA – Analytic Framework SHERPA Components HUE HBASE PIG HIVE Data Sets GEO Enrichment S 30 days 61,764,205 netflows T 30 days 1,065,977 SNORT SQOOP Port Enrichment A T 30 days 4,065,977 Firewall Protocol Enrichment S Packet Data HDFS Firewall Logs ZOOKEEPER Netflow Data Application Server Logging User Logging IDS/IPS Logs sink sink sink sink sink sink User App Netflow Packet Firewall IDS/IPS Logging Server Logging Capture Logs Logs Hadoop World 2011 Logging cloudera Six3 Systems

Develop and Deploy Analytics Risk Potential Correlate Perform Index Report Analytic Results Enrichment Flow Characterization Analytic Analytic Runtime Environment Health & Analytic CORE Data Services Job Control SHERPA Status Registry Services SDK SHERPA – Analytic FrameworkHadoop World 2011 cloudera Six3 Systems

SHERPASURFING Toolkit • FLUME Sinks, Decorators • HBASE Object Definitions • Multiple forms of Enrichment • SHERPA Developers Guide and Cookbook • Two Sample Analytics • Enterprise Analytic FrameworkHadoop World 2011 cloudera Six3 Systems

ConclusionsHadoop World 2011 cloudera Six3 Systems

The Wrap-up • The threat is very real, well funded and determined • The problem has an incredible often hidden impact • Apache Hadoop stack provide an effective foundation • SHERPA solution builds on that stack • Provides a framework for Cyber Security AnalyticsHadoop World 2011 cloudera Six3 Systems

sherpasurfing@gmail.com QUESTIONS?“Imagination is more important than knowledge. For knowledge is limited to all wenow know and understand, while imagination embraces the entire world, and allthere ever will be to know and understand.”~Albert EinsteinHadoop World 2011 cloudera Six3 Systems

http://www.cloudera.com	435
http://www.linkedin.com	8
http://blog.cloudera.com	4
http://cloudera.brian.dev	2
http://cloudera.louddog.net	2
http://cloudera.matt.dev	2

Hadoop World 2011: Sherpasurfing - Wayne Wheeles

by Cloudera, Inc. on Nov 10, 2011

Accessibility

Categories

Upload Details

Usage Rights

6 Embeds 453

Statistics

Hadoop World 2011: Sherpasurfing - Wayne Wheeles Presentation Transcript