Your SlideShare is downloading. ×
Yahoo! Mail antispam - Bay area Hadoop user group
Yahoo! Mail antispam - Bay area Hadoop user group
Yahoo! Mail antispam - Bay area Hadoop user group
Yahoo! Mail antispam - Bay area Hadoop user group
Yahoo! Mail antispam - Bay area Hadoop user group
Yahoo! Mail antispam - Bay area Hadoop user group
Yahoo! Mail antispam - Bay area Hadoop user group
Yahoo! Mail antispam - Bay area Hadoop user group
Yahoo! Mail antispam - Bay area Hadoop user group
Yahoo! Mail antispam - Bay area Hadoop user group
Yahoo! Mail antispam - Bay area Hadoop user group
Yahoo! Mail antispam - Bay area Hadoop user group
Yahoo! Mail antispam - Bay area Hadoop user group
Yahoo! Mail antispam - Bay area Hadoop user group
Yahoo! Mail antispam - Bay area Hadoop user group
Yahoo! Mail antispam - Bay area Hadoop user group
Yahoo! Mail antispam - Bay area Hadoop user group
Yahoo! Mail antispam - Bay area Hadoop user group
Yahoo! Mail antispam - Bay area Hadoop user group
Yahoo! Mail antispam - Bay area Hadoop user group
Yahoo! Mail antispam - Bay area Hadoop user group
Yahoo! Mail antispam - Bay area Hadoop user group
Yahoo! Mail antispam - Bay area Hadoop user group
Yahoo! Mail antispam - Bay area Hadoop user group
Yahoo! Mail antispam - Bay area Hadoop user group
Yahoo! Mail antispam - Bay area Hadoop user group
Yahoo! Mail antispam - Bay area Hadoop user group
Yahoo! Mail antispam - Bay area Hadoop user group
Yahoo! Mail antispam - Bay area Hadoop user group
Yahoo! Mail antispam - Bay area Hadoop user group
Yahoo! Mail antispam - Bay area Hadoop user group
Yahoo! Mail antispam - Bay area Hadoop user group
Yahoo! Mail antispam - Bay area Hadoop user group
Yahoo! Mail antispam - Bay area Hadoop user group
Yahoo! Mail antispam - Bay area Hadoop user group
Yahoo! Mail antispam - Bay area Hadoop user group
Yahoo! Mail antispam - Bay area Hadoop user group
Yahoo! Mail antispam - Bay area Hadoop user group
Yahoo! Mail antispam - Bay area Hadoop user group
Yahoo! Mail antispam - Bay area Hadoop user group
Yahoo! Mail antispam - Bay area Hadoop user group
Yahoo! Mail antispam - Bay area Hadoop user group
Yahoo! Mail antispam - Bay area Hadoop user group
Yahoo! Mail antispam - Bay area Hadoop user group
Yahoo! Mail antispam - Bay area Hadoop user group
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Yahoo! Mail antispam - Bay area Hadoop user group

15,608

Published on

Published in: Technology
0 Comments
18 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
15,608
On Slideshare
0
From Embeds
0
Number of Embeds
10
Actions
Shares
0
Downloads
253
Comments
0
Likes
18
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide
  • Who knows what Yokai are? <audience poll>Shape-shifters from Japanese mythology. Many other examples, e.g. Proteus, who would tell you the future, but first you had to capture him. Just like the gods, Change shape to avoid capture* vary over IP, vary over content, vary over template features (e.g. document structure, subjects, size entropy)
  • In abuse, these are “shape shifters.”They vary many aspects of the message to avoid detection: IPSubjectContentFor example, these four messages are obviously built from a single template, but changing its shape to avoid capture. How to catch?In the past: + Heuristics & Regex + Dictionary (URLdb) + Invariant metadataChallenges: + slow to write+ difficult to write+ easy to evade
  • Here is a third type of shape-shifting spamFor all of these: attackers have distinct advantage, because they can change most aspects and still get through
  • 1.3 sextillion (1.3e21) variations, almost all can be recognized by human being in milisecondsspammers learned they can change any variable to hide from bulk filtershttp://cockeyed.com/lessons/viagra/viagra.html
  • These bastards… the most despised doctors on the InternetAlmost all pages resolve through numerous HTML/Javascript redirectors to this page
  • Daniel Geer said, there are targets of CHANCE and targets of CHOICE. Small businesses are in the former camp, catching the miscellaneous attacks out there.Increasingly, larger companies are TARGETS OF CHOICE, meaning the bad guys a) specifically tailor their attacks based on known vulnerabilities, and b) use feedback loops to improve the effectiveness of them.
  • This is what a targetted attack profile looks like: After you patch, they almost stop trying
  • One example of such a clearly targeted attack: 400KB of style gibberish embedded in a style sheet, completely throws out our parsersMaybe ASCII art spam, or something else that couldn’t be caught by simple pattern matchingThis is what our filters see: a stream of ASCII that is deliberately using multiple layerse.g. here, a TinyURL redirector, further obfuscated with non-printing HTML, spaces, and CSS chaffTo fight in olden days, hand-written regex to identify a patternOR heuristic on some invariant part of the message. But what is invariant? dozens of TinyURL clonesdozens of HTML and CSS tricks2^32 IP addressesinfinite FROM addressesinfinite SUBJECT lines…
  • Sent by botnetsThis is Reactor Mailer; controlled Srizbi from the McColo datacenters until Nov 2008This is the template for Stormbot; notice it has control variables for all the settingsWhile most of these came in through SMTP port 25, now they are increasingly hitting HTTP and port 80
  • Historically, POINT SOLUTIONS address each problem individuallyregexheuristicWouldn’t this be better if this guy could use more than one finger at a time?Something is *almost over the limit* along one dimension and *almost over the limit* along another.Message from IP that sends 80% good mail, with tinyurl that we don’t recognize, that was addressed to 40 people.*PRIOR PROBABILITY**COMPOSITE SCORE*
  • Scale forces simplistic architectures; Feedback based architectures always lag behind the spam campaignFeedback also has many segments;- Personal preference spam: “I didn’t like this week’s Amazon gold box deals but I liked last week’s messages from Amazon”- Annoyance emails from legitimate bulk mailers: “This coupon is coming far too often these days”-Listserver spam: “This finance group - Newsletter messages that are no longer interesting to the user: “Gosh I am so not into that band any more”sometimes sends me stock spam”Traffic to a small enterprise domains can be restricted with firewall rules etc but large free mail provider traffic is full of corner casesCompounding the problem is the fact that adoption of DKIM and SPF has been slow, especially internationally and in emerging economies.But make no mistake, some of these spammers are very cleverIts more fruitful to target yahoo or google than to build a generic spam engine
  • Lets looks at what is in place right now in terms of an architecture; Most large scale systems have some components from gen1 technologiesProvide attack mitigation and operational flexibility, highly explainable. Not durable, expensive to keep pace with fast morphing spamProprietary implementations, not very scalable, steep learning curveReactive and usually late
  • Two ways this has been solved in the past: Machine management…Both systems, because of scale, were limited to looking at small pieces of data – an IP, a URL, etc.
  • In this talk we’ll introduce Hadoop, an open-source grid computing environment with applications to fighting abuse. We’ll talk about how Hadoop can be applied to polymorphic spam and abuseAbout three years ago, Doug Cutting released version 0.15 of Hadoop, an open-source platform inspired by Google’s proprietary Map:Reduce algorithm“Supercomputer” – petabytes of storage, terabytes of RAM allow “needle in the haystack” even at Y!Mail scalehundred of featureshundreds of billions of recordstrends buried in global data
  • Hadoop is the most prevalentAlso “Ngrid” and “Sun’s GridEngine” are other alternatives
  • Input data format is application-specific, specified by the user Output is a set of <key,value> pairs User expresses algorithm using two functionsMap is applied on the input data and produces a list of intermediate <key,value> pairs Reduce is applied to all intermediate pairs with the same key. It typically performs some kind of merging operation and produces zero or more output pairs Finally, output pairs are sorted by their key value
  • Toy exampleProvide some insight into what a map reduce program looks like, looks very much like unix command line
  • Java code to highlight the mapper, mapper simply adds each word to a set and emits a count of 1 for each time the word is seen
  • The reducer simply sums the values for each word, draw attention to line 32While this is a toy example, it should give a fair idea about how to structure a problem to be solvable by map reduce. The key takeaway is that writing even native map reduce programs can be quite simple and executing it even simpler
  • Take the audience progressively through more and more sophisticated applications, starting from basic reporting and ending in outbound spammer analysis based on SWARM features
  • Knowing the accurancy of your SVM/Bayes classifier puts you in no better situation to ask and answer what type of spam is leaking; and we know spammers are constantly probing80% of the spam/content classification problem is in smart feature engineering
  • Lets looks at how our/Yahoo’s platform looks like Perl programs for feature engineering make it very easy and flexibleHadoop with its pig support is already well suited as a platform for adhoc data analysisFor deep data mining, open source mahout
  • We will look at the hadoop is four different settings;
  • * In antispam, these basic reports combined with human review form a barrier against highly directed attacks that exploit system weaknesses* Note how easy it is to slice and dice your data and write fairly sophisticated reports using pig/streaming. It is critical in antispam systems that the reporting platform be flexible and provide a lot of expressive power, hadoop and pig achieve that.*
  • Previous such queries were against small samples, now we can do it against the full data set and get highly accurate results in a very short amount of timeAlternate architectures such as OLAP are too expensive at this scale
  • * Pig is a data flow specification language. Its like SQL but unlike SQL it is better suited for data flow control. * In antispam, these basic reports combined with human review form a barrier against highly directed attacks that exploit system weaknessesNote how easy it is to slice and dice your data and write fairly sophisticated reports using pig. It is critical in antispam systems that the reporting platform be flexible and provide a lot of expressive power, hadoop and pig achieve that.*
  • -- People who bought eggs also bought bread
  • * We ran frequent itemset on one day’s spam votes, the results are striking.* Notice in the above example how the same campaign [the same FROMUSER] is being managed with different templates for subjects and URLs and is also originating from different IPs* Others records in the background are the result of the freq itemset mining algorithm as well and map very closely with spam campaigns.
  • Develop a bipartite graph of users and the IPs they vote fromSquaring of the graph give rise to connected componentWeight of the connected component is a measured by the number of vertices that share the component.
  • GamingIPs are IPs that the spammers try to whitelist in advanceDetected them by extending the connected component view on Ips the notspam is voted on
  • The results are quite spectacular!! There is a massive amount of “gaming” going on with “notspam feedback” and there are only a handful of Ips that are doing this. There are a large number of smaller components not shown in the results above
  • The results are less stronger – notice the two smaller weaker clusters in row 3 and 4The big takeway is that such unsupervised matching algorithms are going to be extremely power amplifiers of signals and can be used to rapidly separate out noise from signal.Imagine this being applied on traffic with more items such as IPs, message subjects, size of messages, fuzzy signatures etc.
  • We encourage and invite others to try hadoop in anti spam and anti abuse architectures and share their experiences with us.
  • Three users known badsame IP leads to new cookiesame cookie leads to new birthdayetc.*AMPLIFICATION OF SMALL SIGNAL*
  • Transcript

    • 1. Yokai Versus the ElephantHadoop and the Fight Against Shape-Shifting Spam
      VishwanathRamarao & Mark Risher
      Yahoo! Mail
    • 2. © SHMorgan - www.obakemono.com
    • 3. AGENDA
      3
      Shape-shifting spam
      Antispam Origins
      Hadoop Algorithms
      Applications to Security
      Resources for Implementers
    • 4.
    • 5. 5
    • 6. 6
      http:/<!--gmail.com-->/f915fde2cf53df18<!--uc22wddprm-->.li<!--cf997b28e-->gh<!--PdNKLr-->
      tt<!---kxnd2itipuvd.yahoo.com-->o<!--ju1j8V-->
      p<!--vrgxetdcnubslgacvc-->b<!--OsLaWIv-->o<!--_qsgsnnjuf1m@vkvriskrgavzxjovbqg.net-->dy<!--in7oouvxfrg7ax-->.com]*!}v}]along especially consecutive important dmvfu
      <!--gmail.com-->
    • 7. 7
    • 8. 8
      1,300,925,111,156,286,160,896
      (http://bit.ly/cpOyLi)
    • 9.
    • 10. 10
    • 11. Typical attack/response profile
      11
      Rule change
      (1/23@01:15)
    • 12. MORE YOKAI - TARGETED ATTACKS
      <style>mechanic CC0066 getimage 3A00 lectroniques repertoires spiel proscribing ammonoid 10110 radiobuttontelefoons Jermaine iesaporitoroshan 3026 janatatrennungpalillos toughest ncapitolecalzado 20200 Omnimedia collective saudadedizaines 205px hardener elongating InvasionofyourprivacyPersonnalftsbedingungenMontanerprozacSerpellfcardbvh capacitate 12502 courtship kiranjiutroligt transducer tyee Delhaize clueless toffee nnioZoapochino sterns 622 Verordnung carbons waterresistant assessing footerTextperrine url0 potatoes 999933 Rightmove positively thmb closer secures Amarillo suffer 314992 32599 8849 GJ initialling cockleshell JTA Justiaguardo jibes Chubb inflammatory iteration granfaldasseoir considerations 692px treasured Allotransplantationtwoyearsappx Bowers doorgeven 1487 bigpicture repeatedly Popp MPEG4 webbsidaliefdeVoeding Elena Kernighan sternway laggardly Zwischendurch commons equis sewing f17 apadrinasareiniqueslugoquotedblbayr 3500 CI addressee optativelygazzetta 616px mingus 23238 PhotoLink desuetude tofu keychains molding redevelopment stucco deltage astrology2 thumbscrews probablemente 700g rnsfuseactionrepristaires restraint manchettestrendlineseffectuedespatchMinskyestadual doses danbrown Muenster jind7n7 smashes gourmandesashantisentants rows kyk coated Incontournablescoincidenjspa stalker CDS contienen expletives s8 eof replenishing puyalluppratosondravalidarorientale sonnets steamer Niwangoacrocentric dozens elr tempting poing jails ingredi Sep3 misdirection vested tecniciconciertos dear martini 3D35 MBR DNAME 2650 violation Egyptiin NCR sposoriss hl 12450 connectors circumcision transform CFA employeur 153 comunicazioni miner 19905 citronella PlissierHellmich Randall CaradonnaspringaregistradahauptEntran 3060 Rochin capacitor sotol 3413 smirk interditeServicePoint capabilities bouncefeeLinkov 3Dg auntie OSP CaeciliaPlatzierung wrangler pisosbanlieueDaniellaenderleisraelprofessionnellessusto 39800 Espanaplena radian antic!...........................200KB……….
      </style>
      <center><a href="http://ivywhere.info/52210088504303.hrmj.1/285/1000/1006/1000/1237976a102c0176c7b3fb3164f83590.html">Please Click Here if You Can't See Images<br><imgsrc="http://ivywhere.info/images/usacpm1.jpg" border="0"></a><br><a href="http://ivywhere.info/52210088504303.hrmj.1/40106/1000/1000/1000/a.html"><imgsrc="http://ivywhere.info/images/usacpm2.jpg" border="0"></a><br><a href="http://ivywhere.info/gp.html"><imgsrc="http://ivywhere.info/images/please2.jpg" border="0"></a><br>
      12
      [400kb…]
      <center><a href="http://corfair.info/52210088504303.hrmj.1/129286/1000/1006/1000/d1c7b1fa06980b08bf9b3a9c14844623.html">Please Click Here if You Can't See Images<br><imgsrc="http://corfair.info/images/ivblg1.jpg" border="0"></a><br><a href="http://corfair.info/52210088504303.hrmj.1/40126/1000/1000/1000/a.html"><imgsrc="http://corfair.info/images/ivblg2.jpg" border="0"></a><br><a href="http://corfair.info/gp.html"><imgsrc="http://corfair.info/images/please2.jpg" border="0"></a><br>
    • 13.
    • 14. 14
    • 15. Why is the ANTISPAM PROBLEM hard
      Scale of the problem; 25B Connections, 5B deliveries, 450M mailboxes
      User feedback is often late, noisy and not always actionable
      Large, diverse stream of legitimate traffic that looks like spam
      Slow adoption of authentication technologies like DKIM and SPF
      Spammers are clever; target and specialize attacks
      Rapidly changing spam campaigns with a large bot controlled IP base; large variations even within a single campaign
      A significant percentage of spam comes from large ESPs like Hotmail, Google and Yahoo
      15
    • 16. Generation 1: Manual management layer
      Heuristics, blocks, blacklists
      Provide attack mitigation and operational flexibility, highly explainable.
      Not durable, expensive to keep pace with fast morphing spam
      Ad hoc queries
      Proprietary implementations, not very scalable, steep learning curve
      Reactive and usually late
      16
    • 17. Generation 2: Machine Management Layer
      Online reputation models
      Simple, mostly scoring/counter/ratio based models
      Highly scalable due the absence of any state/memory
      Generalize too broadly, lack expressive power
      Batch trained reputation models
      Typically digested memory based hashing or machine learning models
      Difficult to implement and due to the need for labeled examples scale well only moderately
      Slow to update and learn, lack explainability, limited operational control
      17
    • 18.
    • 19. distributed computing paradigm
      19
      Map:Reduce + distributed storage:
      • Simplicity of online, stateless models
      • 20. Expressiveness of offline analysis
      • 21. Ease of management
    • the map:reduce paradigm
      Input data format is application-specific, specified by the user
      Output is a set of <key,value> pairs
      User expresses algorithm using two functions
      Map is applied on the input data and produces a list of intermediate <key,value> pairs
      Reduce is applied to all intermediate pairs with the same key. It typically performs some kind of merging operation and produces zero or more output pairs
      Finally, output pairs are sorted by their key value
      20
    • 22. the map:reduce paradigm
      21
      Mapper
      <k1,v1>
      Mapper
      <k1,{v1,v3}>
      <k2,v2>
      Reducer
      <k2,v2>
      <k1,W1>
      Mapper
      <k1,v3>
    • 23. A SIMPLE MAP:REDUCE EXAMPLE
      $ bin/hadoopdfs -cat /usr/joe/wordcount/input/file01
      Hello World Bye World
      $ bin/hadoopdfs -cat /usr/joe/wordcount/input/file02
      Hello Hadoop Goodbye Hadoop
      // Split up input files (MAP), iterate over chunks, reassemble results (REDUCE)
      $ bin/hadoop jar /usr/joe/wordcount.jarorg.myorg.WordCount /usr/joe/wordcount/input /usr/joe/wordcount/output
      $ bin/hadoopdfs -cat /usr/joe/wordcount/output/part-00000
      Bye 1
      Goodbye 1
      Hadoop 2
      Hello 2
      World 2
      22
    • 24. a simple map:reduce example (bit.ly/bdyi0l)
      18. public void map(LongWritable key, Text value, OutputCollector<Text, IntWritable> output, Reporter reporter) throws IOException {
      19. String line = value.toString();
      20. StringTokenizertokenizer = new StringTokenizer(line);
      21. while (tokenizer.hasMoreTokens()) {
      22. word.set(tokenizer.nextToken());
      23. output.collect(word, one);
      24. }
      25. }
      23
    • 25. a simple map:reduce example (bit.ly/bdyi0l)
      28. public static class Reduce extends MapReduceBase implements Reducer<Text, IntWritable, Text, IntWritable> {
      29. public void reduce(Text key, Iterator<IntWritable> values, OutputCollector<Text, IntWritable> output, Reporter reporter) throws IOException {
      30. int sum = 0;
      31. while (values.hasNext()) {
      32. sum += values.next().get();
      33. }
      34. output.collect(key, new IntWritable(sum));
      24
    • 26. Applications
      &
      Outcomes
      25
    • 27. Lets REVIEW OUR DESIGN GOALs AGAIN
      Classifiers are notorious for lack of explainability
      Engineers and analysts needs to know what the classifier is missing
      Engineers and analysts need to know about emerging threats
      Analysts need “canned” reports along interesting dimensions
      Machines need smart feature engineering
      Develop a scalable system to provide deep insight into spammer campaigns
      Double up as a platform for standard reporting
      Also double up as a platform for adhoc analysis and data probing
      Signal amplification and smart feature extraction platform
      26
    • 28. Our ANTISPAM ANALYTIC PLATFORM
      Hadoop: Implements map reduce, written in Java but supports many other languages including Perl and C++ using the streaming interface
      Feature engineering with small simple Perl programs for data extraction and transformation
      SQL-like “Pig” programming language for data analysis and management
      Mahout: data mining libraries that provide shrink- wrapped, scalable, sophisticated algorithms
      Other proprietary algorithms and frameworks for specialized tasks
      27
    • 29. Various ASPECTS of A GRID DRIVEN SOLUTION
      Standard reporting
      Ad hoc querying
      Campaign discovery from spam feedback using frequent item set mining
      “Gaming” detection in notspam feedback using connected components
      28
    • 30. Top SPAMMY DOMAINS REPORT FOR 01/15/2010
      29
      key:noreply.amateurmatch.com|value:1164
      key:goodmere.info|value:896
      key:marketing.meredith.com|value:1078
      key:verizon.net|value:822
      key:reply.mb00.net|value:980
      key:insideapple.apple.com|value:1094
      key:facebookappmail.com|value:882
      key:mydailymoment.com|value:849
      key:thetwilightsaga.com|value:4671
      key:adknowledgemailer6.com|value:859
      key:freedollarspro.info|value:1164
      key:smartreachmedia.com|value:1074
      key:yahoo.es|value:877
      key:ecomasher.com|value:1197
      key:leasetrade-statusupdates.com|value:951
      key:noreply.amateurmatch.comvalue:1164
    • 31. AD HOC queries for ANTISPAM research
      Identify domains that had few spam votes in the previous time window but have a high number of spam votes today
      All IPs in the last hour that sent a particular URL pattern…or that sent any unknown URL >500 times
      Which domains/IPs suddenly increased their sending volume after a positive reputation change
      Which FROM addresses exhibit low message size entropy
      All messages that had nothing but a URL and the domain of the URL had low page rank
      30
    • 32. AD HOC QUERIES - Anatomy of a PIG QUERY
      --- This includes some basic string functions, including splitting a string on the '@' character
      register /homes/jpujara/pig_scripts/string.jar;
      define splitEmail string.Tokenize('2','@');
      --- Load up some data - incoming messages at a date and time, and our trusted user database
      MESSAGES = load '/projects/antispam/mta_feature_logs/$date*/*/*-$time*' using com.yahoo.ymail.pigfunctions.AsStorage('__record_key__,firstrcpt,mailfrom') as (mid:chararray,to:chararray,from:chararray);
      USERS = load '/projects/antispam/TrustedUser.bz2' using com.yahoo.ymail.pigfunctions.AsStorage('user,t') as (user:chararray,trusted:int);
      --- Split the e-mail addresses into user+domain and generate the appropriate user-id for yahoo users and partners
      EXPLODED_MESSAGES = FOREACH MESSAGES GENERATE to,FLATTEN(splitEmail(to)) as (user,udomain),FLATTEN(splitEmail(from)) as (sender,sdomain);
      YAHOO_MESSAGES = FOREACH EXPLODED_MESSAGES GENERATE (udomain MATCHES '.*yahoo.*' ? user : to ) as yuser,sdomain;
      31
      --- Combine the message and sender domains with the trusted user data and select only trusted messages
      YAHOO_MESSAGES_TRUST = JOIN YAHOO_MESSAGES by yuser, USERS by user;
      TRUSTED_MESSAGES = FILTER YAHOO_MESSAGES_TRUST by trusted > 0;
      --- Group by domain, and generate a count, order by descending count
      DOMAIN_GROUPS = GROUP TRUSTED_MESSAGES by sdomain;
      DOMAIN_GROUPS_COUNT = FOREACH DOMAIN_GROUPS GENERATE group,COUNT(TRUSTED_MESSAGES) as count;
      DOMAIN_GROUPS_ORDER = ORDER DOMAIN_GROUPS_COUNT by count DESC;
      --- Output the results
      STORE DOMAIN_GROUPS_ORDER into '$targetdir/topDomains';
    • 33. CAMPAIGN Discovery in SPAM Feedback
      Frequent Itemset Mining
      Classical method
      Research interesting relationships between variables in a large database
      Primarily applied for market basket analysis
      Many good implementations
      APRIORI
      Easy to implement
      Parallelizes moderately well but bottlenecks for extremely large data sets
      Not very efficient with the number scans
      ECLAT
      Parallelizes easily
      Amenable to a good grid implementation
      Fewer scans of the dataset
      Parallel FP GROWTH
      Designed explicitly for systems like hadoop
      Implemented in Mahout 0.2
      32
    • 34. Frequent item set – example dataset
      33
    • 35. Frequent ITEMSET MINING
      34
      Slide Courtsey: dortmund.de
    • 36. Frequent itemset MINING on ONE DAY’s SPAM REPORTS
      9 2595 (IPTYPE:none,FROMUSER:sales,SUBJ:It's Important You Know,FROMDOM:dappercom.info,URL:dappercom.info,ip_D:66.206.14.77,)
      9 2457 (IPTYPE:none,FROMUSER:sales,SUBJ:Save On Costly Repairs,FROMDOM:aftermoon.info,URL:aftermoon.info,ip_D:66.206.14.78,)
      9 2447 (IPTYPE:none,FROMUSER:sales,SUBJ:Car-Dealers-Compete-On-New-Vehicles,FROMDOM:sherge.info,URL:sherge.info,ip_D:66.206.25.227,)
      9 2432 (IPTYPE:none,FROMUSER:sales,SUBJ:January 18th: CreditReport Update,FROMDOM:zaninte.info,URL:zaninte.info,ip_D:66.206.25.227,)
      9 2376 (IPTYPE:none,FROMUSER:health,SUBJ:Finally. Coverage for the whole family,FROMDOM:fiatchimera.com,URL:articulatedispirit.com,ip_D:216.218.201.149,)
      9 2184 (IPTYPE:none,FROMUSER:health,SUBJ:Finally. Coverage for the whole family,FROMDOM:fiatchimera.com,URL:stratagemnepheligenous.com,ip_D:216.218.201.149,)
      9 1990 (IPTYPE:none,FROMUSER:sales,SUBJ:Closeout 2008-2009-2010 New Cars,FROMDOM:sastlg.info,URL:sastlg.info,ip_D:66.206.25.227,)
      9 1899 (IPTYPE:none,FROMUSER:sales,FROMDOM:brunhil.info,SUBJ:700-CreditScore-What-Is-Yours?,URL:brunhil.info,ip_D:66.206.25.227,)
      9 1743 (IPTYPE:none,FROMUSER:sales,SUBJ:Now exercise can be fun,FROMDOM:accordpac.info,URL:accordpac.info,ip_D:66.206.14.78,)
      9 1706 (IPTYPE:none,FROMUSER:sales,SUBJ:Closeout 2008-2009-2010 New Cars,FROMDOM:rionel.info,URL:rionel.info,ip_D:66.206.25.227,)
      9 1693 (IPTYPE:none,FROMUSER:sales,SUBJ:January 18th: CreditReport Update,FROMDOM:astroom.info,URL:astroom.info,ip_D:66.206.25.227,)
      9 1689 (IPTYPE:none,FROMUSER:sales,SUBJ:eBay: Work@Home w/Solid-Income-Strategies,FROMDOM:stamine.info,URL:stamine.info,ip_D:66.165.232.203,)
      35
      2432 (IPTYPE:none,FROMUSER:sales,SUBJ:January 18th: CreditReportUpdate,FROMDOM:zaninte.info,URL:zaninte.info, ip_D:66.206.25.227,)
      2447 (IPTYPE:none,FROMUSER:sales,SUBJ:Car-Dealers-Compete-On-New-Vehicles,FROMDOM:sherge.info,URL:sherge.info,
      ip_D:66.206.25.227,)
    • 37. Gaming DETECTION in NOTSPAM FEEDBACK
      • Spammers instrument accounts to vote “not spam” on emails that they send
      • 38. Delays classification of spamming IP addresses
      Throws off the classifiers if the feedback is not filtered well
      Model the problem as a bipartite graph
      Well known model for matching algorithms
      Broadly applied in various fields like coding theory
      A graph whose vertices are disjoint form disjoint sets U,V
      There is an edge connecting every U to a vertex in V
      36
    • 39. Connected COMPONETS - EXPLAINED
      Y1 = Yahoo user 1, Y2 = Yahoo user 2
      IP1 = IP address of the host Y1 “voted” notspam from
      37
      y1
      IP1
      y1
      SQUARING
      weight = 2
      y1
      IP2
      y1
    • 40. Connected COMPONENTS for “GAMING” DETECTION
      38
      Set of IPs/YIDs used
      exclusively for
      voting notspam
      Set of (likely new)
      spamming IPs which
      are “worth” voting for
      y1
      IP3
      IP1
      y2
      IP4
      IP2
      y3
      Set of
      “voted on” IPs
      Set of
      “voted from” IPs
      Set of Yahoo IDs
      voting notspam
    • 41. Connected Components - RESULTS
      39
      - Connnected components for IPsnotspam was voted from
    • 42. Connected components - results
      40
      - Connnected components for IPsnotspam was voted on
    • 43. CONCLUSIONS
      We have had success leveraging parallel, stateful algorithms on grid systems to keep pace with polymorphic spam that evade traditional analysis and algorithms
      Frequent Itemset Mining rapidly identifies cohesive campaigns in ISSPAM feedback
      Connected Components amplifies weak signals in gamed NOTSPAM feedback and helps separate signal from noise in the feedback
      Grid system based analysis platforms may be broadly applicable across the security domain
      41
    • 44. Apply Slide
      Download Hadoop distribution
      http://hadoop.apache.org
      Try out Pig on standalone, single Linux box
      Identify source data to aggregate
      Start simple: IP patterns across web access logs
      Begin with offline aggregation; yesterday’s attacks still interesting
      Read Connected Components and Frequent Itemset Mining papers
      Stop looking for a single, invariant “tell” – far too costly
      Start thinking about co-occurrence of innocuous features
      42
    • 45. Resources for implementers
      Hadoop setup, documentation and resources
      http://hadoop.apache.org/
      Pig documentation and resources
      http://hadoop.apache.org/pig/
      Mahout documentation and resources
      http://lucene.apache.org/mahout/
      Frequent itemset mining implementation repository
      http://fimi.cs.helsinki.fi/src/
      Connected components description
      [link not yet live]
      Ranger, Raghuraman, Penmetsa, Bradski, and Kozyrakis. Evaluating MapReduce for Multi-core and Multiprocessor Systems. In HPCA 2007
      43
    • 46.
    • 47. Connected COMPONENTS
      45

    ×