Your SlideShare is downloading. ×
0
Yahoo! Mail antispam - Bay area Hadoop user group
Yahoo! Mail antispam - Bay area Hadoop user group
Yahoo! Mail antispam - Bay area Hadoop user group
Yahoo! Mail antispam - Bay area Hadoop user group
Yahoo! Mail antispam - Bay area Hadoop user group
Yahoo! Mail antispam - Bay area Hadoop user group
Yahoo! Mail antispam - Bay area Hadoop user group
Yahoo! Mail antispam - Bay area Hadoop user group
Yahoo! Mail antispam - Bay area Hadoop user group
Yahoo! Mail antispam - Bay area Hadoop user group
Yahoo! Mail antispam - Bay area Hadoop user group
Yahoo! Mail antispam - Bay area Hadoop user group
Yahoo! Mail antispam - Bay area Hadoop user group
Yahoo! Mail antispam - Bay area Hadoop user group
Yahoo! Mail antispam - Bay area Hadoop user group
Yahoo! Mail antispam - Bay area Hadoop user group
Yahoo! Mail antispam - Bay area Hadoop user group
Yahoo! Mail antispam - Bay area Hadoop user group
Yahoo! Mail antispam - Bay area Hadoop user group
Yahoo! Mail antispam - Bay area Hadoop user group
Yahoo! Mail antispam - Bay area Hadoop user group
Yahoo! Mail antispam - Bay area Hadoop user group
Yahoo! Mail antispam - Bay area Hadoop user group
Yahoo! Mail antispam - Bay area Hadoop user group
Yahoo! Mail antispam - Bay area Hadoop user group
Yahoo! Mail antispam - Bay area Hadoop user group
Yahoo! Mail antispam - Bay area Hadoop user group
Yahoo! Mail antispam - Bay area Hadoop user group
Yahoo! Mail antispam - Bay area Hadoop user group
Yahoo! Mail antispam - Bay area Hadoop user group
Yahoo! Mail antispam - Bay area Hadoop user group
Yahoo! Mail antispam - Bay area Hadoop user group
Yahoo! Mail antispam - Bay area Hadoop user group
Yahoo! Mail antispam - Bay area Hadoop user group
Yahoo! Mail antispam - Bay area Hadoop user group
Yahoo! Mail antispam - Bay area Hadoop user group
Yahoo! Mail antispam - Bay area Hadoop user group
Yahoo! Mail antispam - Bay area Hadoop user group
Yahoo! Mail antispam - Bay area Hadoop user group
Yahoo! Mail antispam - Bay area Hadoop user group
Yahoo! Mail antispam - Bay area Hadoop user group
Yahoo! Mail antispam - Bay area Hadoop user group
Yahoo! Mail antispam - Bay area Hadoop user group
Yahoo! Mail antispam - Bay area Hadoop user group
Yahoo! Mail antispam - Bay area Hadoop user group
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Yahoo! Mail antispam - Bay area Hadoop user group

15,809

Published on

Published in: Technology
0 Comments
18 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
15,809
On Slideshare
0
From Embeds
0
Number of Embeds
10
Actions
Shares
0
Downloads
254
Comments
0
Likes
18
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide
  • Who knows what Yokai are? <audience poll>Shape-shifters from Japanese mythology. Many other examples, e.g. Proteus, who would tell you the future, but first you had to capture him. Just like the gods, Change shape to avoid capture* vary over IP, vary over content, vary over template features (e.g. document structure, subjects, size entropy)
  • In abuse, these are “shape shifters.”They vary many aspects of the message to avoid detection: IPSubjectContentFor example, these four messages are obviously built from a single template, but changing its shape to avoid capture. How to catch?In the past: + Heuristics & Regex + Dictionary (URLdb) + Invariant metadataChallenges: + slow to write+ difficult to write+ easy to evade
  • Here is a third type of shape-shifting spamFor all of these: attackers have distinct advantage, because they can change most aspects and still get through
  • 1.3 sextillion (1.3e21) variations, almost all can be recognized by human being in milisecondsspammers learned they can change any variable to hide from bulk filtershttp://cockeyed.com/lessons/viagra/viagra.html
  • These bastards… the most despised doctors on the InternetAlmost all pages resolve through numerous HTML/Javascript redirectors to this page
  • Daniel Geer said, there are targets of CHANCE and targets of CHOICE. Small businesses are in the former camp, catching the miscellaneous attacks out there.Increasingly, larger companies are TARGETS OF CHOICE, meaning the bad guys a) specifically tailor their attacks based on known vulnerabilities, and b) use feedback loops to improve the effectiveness of them.
  • This is what a targetted attack profile looks like: After you patch, they almost stop trying
  • One example of such a clearly targeted attack: 400KB of style gibberish embedded in a style sheet, completely throws out our parsersMaybe ASCII art spam, or something else that couldn’t be caught by simple pattern matchingThis is what our filters see: a stream of ASCII that is deliberately using multiple layerse.g. here, a TinyURL redirector, further obfuscated with non-printing HTML, spaces, and CSS chaffTo fight in olden days, hand-written regex to identify a patternOR heuristic on some invariant part of the message. But what is invariant? dozens of TinyURL clonesdozens of HTML and CSS tricks2^32 IP addressesinfinite FROM addressesinfinite SUBJECT lines…
  • Sent by botnetsThis is Reactor Mailer; controlled Srizbi from the McColo datacenters until Nov 2008This is the template for Stormbot; notice it has control variables for all the settingsWhile most of these came in through SMTP port 25, now they are increasingly hitting HTTP and port 80
  • Historically, POINT SOLUTIONS address each problem individuallyregexheuristicWouldn’t this be better if this guy could use more than one finger at a time?Something is *almost over the limit* along one dimension and *almost over the limit* along another.Message from IP that sends 80% good mail, with tinyurl that we don’t recognize, that was addressed to 40 people.*PRIOR PROBABILITY**COMPOSITE SCORE*
  • Scale forces simplistic architectures; Feedback based architectures always lag behind the spam campaignFeedback also has many segments;- Personal preference spam: “I didn’t like this week’s Amazon gold box deals but I liked last week’s messages from Amazon”- Annoyance emails from legitimate bulk mailers: “This coupon is coming far too often these days”-Listserver spam: “This finance group - Newsletter messages that are no longer interesting to the user: “Gosh I am so not into that band any more”sometimes sends me stock spam”Traffic to a small enterprise domains can be restricted with firewall rules etc but large free mail provider traffic is full of corner casesCompounding the problem is the fact that adoption of DKIM and SPF has been slow, especially internationally and in emerging economies.But make no mistake, some of these spammers are very cleverIts more fruitful to target yahoo or google than to build a generic spam engine
  • Lets looks at what is in place right now in terms of an architecture; Most large scale systems have some components from gen1 technologiesProvide attack mitigation and operational flexibility, highly explainable. Not durable, expensive to keep pace with fast morphing spamProprietary implementations, not very scalable, steep learning curveReactive and usually late
  • Two ways this has been solved in the past: Machine management…Both systems, because of scale, were limited to looking at small pieces of data – an IP, a URL, etc.
  • In this talk we’ll introduce Hadoop, an open-source grid computing environment with applications to fighting abuse. We’ll talk about how Hadoop can be applied to polymorphic spam and abuseAbout three years ago, Doug Cutting released version 0.15 of Hadoop, an open-source platform inspired by Google’s proprietary Map:Reduce algorithm“Supercomputer” – petabytes of storage, terabytes of RAM allow “needle in the haystack” even at Y!Mail scalehundred of featureshundreds of billions of recordstrends buried in global data
  • Hadoop is the most prevalentAlso “Ngrid” and “Sun’s GridEngine” are other alternatives
  • Input data format is application-specific, specified by the user Output is a set of <key,value> pairs User expresses algorithm using two functionsMap is applied on the input data and produces a list of intermediate <key,value> pairs Reduce is applied to all intermediate pairs with the same key. It typically performs some kind of merging operation and produces zero or more output pairs Finally, output pairs are sorted by their key value
  • Toy exampleProvide some insight into what a map reduce program looks like, looks very much like unix command line
  • Java code to highlight the mapper, mapper simply adds each word to a set and emits a count of 1 for each time the word is seen
  • The reducer simply sums the values for each word, draw attention to line 32While this is a toy example, it should give a fair idea about how to structure a problem to be solvable by map reduce. The key takeaway is that writing even native map reduce programs can be quite simple and executing it even simpler
  • Take the audience progressively through more and more sophisticated applications, starting from basic reporting and ending in outbound spammer analysis based on SWARM features
  • Knowing the accurancy of your SVM/Bayes classifier puts you in no better situation to ask and answer what type of spam is leaking; and we know spammers are constantly probing80% of the spam/content classification problem is in smart feature engineering
  • Lets looks at how our/Yahoo’s platform looks like Perl programs for feature engineering make it very easy and flexibleHadoop with its pig support is already well suited as a platform for adhoc data analysisFor deep data mining, open source mahout
  • We will look at the hadoop is four different settings;
  • * In antispam, these basic reports combined with human review form a barrier against highly directed attacks that exploit system weaknesses* Note how easy it is to slice and dice your data and write fairly sophisticated reports using pig/streaming. It is critical in antispam systems that the reporting platform be flexible and provide a lot of expressive power, hadoop and pig achieve that.*
  • Previous such queries were against small samples, now we can do it against the full data set and get highly accurate results in a very short amount of timeAlternate architectures such as OLAP are too expensive at this scale
  • * Pig is a data flow specification language. Its like SQL but unlike SQL it is better suited for data flow control. * In antispam, these basic reports combined with human review form a barrier against highly directed attacks that exploit system weaknessesNote how easy it is to slice and dice your data and write fairly sophisticated reports using pig. It is critical in antispam systems that the reporting platform be flexible and provide a lot of expressive power, hadoop and pig achieve that.*
  • -- People who bought eggs also bought bread
  • * We ran frequent itemset on one day’s spam votes, the results are striking.* Notice in the above example how the same campaign [the same FROMUSER] is being managed with different templates for subjects and URLs and is also originating from different IPs* Others records in the background are the result of the freq itemset mining algorithm as well and map very closely with spam campaigns.
  • Develop a bipartite graph of users and the IPs they vote fromSquaring of the graph give rise to connected componentWeight of the connected component is a measured by the number of vertices that share the component.
  • GamingIPs are IPs that the spammers try to whitelist in advanceDetected them by extending the connected component view on Ips the notspam is voted on
  • The results are quite spectacular!! There is a massive amount of “gaming” going on with “notspam feedback” and there are only a handful of Ips that are doing this. There are a large number of smaller components not shown in the results above
  • The results are less stronger – notice the two smaller weaker clusters in row 3 and 4The big takeway is that such unsupervised matching algorithms are going to be extremely power amplifiers of signals and can be used to rapidly separate out noise from signal.Imagine this being applied on traffic with more items such as IPs, message subjects, size of messages, fuzzy signatures etc.
  • We encourage and invite others to try hadoop in anti spam and anti abuse architectures and share their experiences with us.
  • Three users known badsame IP leads to new cookiesame cookie leads to new birthdayetc.*AMPLIFICATION OF SMALL SIGNAL*
  • Transcript

    • 1. Yokai Versus the ElephantHadoop and the Fight Against Shape-Shifting Spam<br />VishwanathRamarao &amp; Mark Risher<br />Yahoo! Mail<br />
    • 2. © SHMorgan - www.obakemono.com<br />
    • 3. AGENDA<br />3<br />Shape-shifting spam<br />Antispam Origins<br />Hadoop Algorithms<br />Applications to Security<br />Resources for Implementers<br />
    • 4.
    • 5. 5<br />
    • 6. 6<br />http:/&lt;!--gmail.com--&gt;/f915fde2cf53df18&lt;!--uc22wddprm--&gt;.li&lt;!--cf997b28e--&gt;gh&lt;!--PdNKLr--&gt;<br />tt&lt;!---kxnd2itipuvd.yahoo.com--&gt;o&lt;!--ju1j8V--&gt;<br />p&lt;!--vrgxetdcnubslgacvc--&gt;b&lt;!--OsLaWIv--&gt;o&lt;!--_qsgsnnjuf1m@vkvriskrgavzxjovbqg.net--&gt;dy&lt;!--in7oouvxfrg7ax--&gt;.com]*!}v}]along especially consecutive important dmvfu<br />&lt;!--gmail.com--&gt;<br />
    • 7. 7<br />
    • 8. 8<br />1,300,925,111,156,286,160,896<br />(http://bit.ly/cpOyLi)<br />
    • 9.
    • 10. 10<br />
    • 11. Typical attack/response profile<br />11<br />Rule change<br />(1/23@01:15)<br />
    • 12. MORE YOKAI - TARGETED ATTACKS<br />&lt;style&gt;mechanic CC0066 getimage 3A00 lectroniques repertoires spiel proscribing ammonoid 10110 radiobuttontelefoons Jermaine iesaporitoroshan 3026 janatatrennungpalillos toughest ncapitolecalzado 20200 Omnimedia collective saudadedizaines 205px hardener elongating InvasionofyourprivacyPersonnalftsbedingungenMontanerprozacSerpellfcardbvh capacitate 12502 courtship kiranjiutroligt transducer tyee Delhaize clueless toffee nnioZoapochino sterns 622 Verordnung carbons waterresistant assessing footerTextperrine url0 potatoes 999933 Rightmove positively thmb closer secures Amarillo suffer 314992 32599 8849 GJ initialling cockleshell JTA Justiaguardo jibes Chubb inflammatory iteration granfaldasseoir considerations 692px treasured Allotransplantationtwoyearsappx Bowers doorgeven 1487 bigpicture repeatedly Popp MPEG4 webbsidaliefdeVoeding Elena Kernighan sternway laggardly Zwischendurch commons equis sewing f17 apadrinasareiniqueslugoquotedblbayr 3500 CI addressee optativelygazzetta 616px mingus 23238 PhotoLink desuetude tofu keychains molding redevelopment stucco deltage astrology2 thumbscrews probablemente 700g rnsfuseactionrepristaires restraint manchettestrendlineseffectuedespatchMinskyestadual doses danbrown Muenster jind7n7 smashes gourmandesashantisentants rows kyk coated Incontournablescoincidenjspa stalker CDS contienen expletives s8 eof replenishing puyalluppratosondravalidarorientale sonnets steamer Niwangoacrocentric dozens elr tempting poing jails ingredi Sep3 misdirection vested tecniciconciertos dear martini 3D35 MBR DNAME 2650 violation Egyptiin NCR sposoriss hl 12450 connectors circumcision transform CFA employeur 153 comunicazioni miner 19905 citronella PlissierHellmich Randall CaradonnaspringaregistradahauptEntran 3060 Rochin capacitor sotol 3413 smirk interditeServicePoint capabilities bouncefeeLinkov 3Dg auntie OSP CaeciliaPlatzierung wrangler pisosbanlieueDaniellaenderleisraelprofessionnellessusto 39800 Espanaplena radian antic!...........................200KB……….<br /> &lt;/style&gt;<br />&lt;center&gt;&lt;a href=&quot;http://ivywhere.info/52210088504303.hrmj.1/285/1000/1006/1000/1237976a102c0176c7b3fb3164f83590.html&quot;&gt;Please Click Here if You Can&apos;t See Images&lt;br&gt;&lt;imgsrc=&quot;http://ivywhere.info/images/usacpm1.jpg&quot; border=&quot;0&quot;&gt;&lt;/a&gt;&lt;br&gt;&lt;a href=&quot;http://ivywhere.info/52210088504303.hrmj.1/40106/1000/1000/1000/a.html&quot;&gt;&lt;imgsrc=&quot;http://ivywhere.info/images/usacpm2.jpg&quot; border=&quot;0&quot;&gt;&lt;/a&gt;&lt;br&gt;&lt;a href=&quot;http://ivywhere.info/gp.html&quot;&gt;&lt;imgsrc=&quot;http://ivywhere.info/images/please2.jpg&quot; border=&quot;0&quot;&gt;&lt;/a&gt;&lt;br&gt;<br />12<br />[400kb…]<br />&lt;center&gt;&lt;a href=&quot;http://corfair.info/52210088504303.hrmj.1/129286/1000/1006/1000/d1c7b1fa06980b08bf9b3a9c14844623.html&quot;&gt;Please Click Here if You Can&apos;t See Images&lt;br&gt;&lt;imgsrc=&quot;http://corfair.info/images/ivblg1.jpg&quot; border=&quot;0&quot;&gt;&lt;/a&gt;&lt;br&gt;&lt;a href=&quot;http://corfair.info/52210088504303.hrmj.1/40126/1000/1000/1000/a.html&quot;&gt;&lt;imgsrc=&quot;http://corfair.info/images/ivblg2.jpg&quot; border=&quot;0&quot;&gt;&lt;/a&gt;&lt;br&gt;&lt;a href=&quot;http://corfair.info/gp.html&quot;&gt;&lt;imgsrc=&quot;http://corfair.info/images/please2.jpg&quot; border=&quot;0&quot;&gt;&lt;/a&gt;&lt;br&gt; <br />
    • 13.
    • 14. 14<br />
    • 15. Why is the ANTISPAM PROBLEM hard<br />Scale of the problem; 25B Connections, 5B deliveries, 450M mailboxes<br />User feedback is often late, noisy and not always actionable <br />Large, diverse stream of legitimate traffic that looks like spam<br />Slow adoption of authentication technologies like DKIM and SPF<br />Spammers are clever; target and specialize attacks <br />Rapidly changing spam campaigns with a large bot controlled IP base; large variations even within a single campaign<br />A significant percentage of spam comes from large ESPs like Hotmail, Google and Yahoo<br />15<br />
    • 16. Generation 1: Manual management layer<br />Heuristics, blocks, blacklists<br />Provide attack mitigation and operational flexibility, highly explainable. <br />Not durable, expensive to keep pace with fast morphing spam<br />Ad hoc queries<br />Proprietary implementations, not very scalable, steep learning curve<br />Reactive and usually late<br />16<br />
    • 17. Generation 2: Machine Management Layer<br />Online reputation models<br />Simple, mostly scoring/counter/ratio based models<br />Highly scalable due the absence of any state/memory<br />Generalize too broadly, lack expressive power<br />Batch trained reputation models<br />Typically digested memory based hashing or machine learning models<br />Difficult to implement and due to the need for labeled examples scale well only moderately<br />Slow to update and learn, lack explainability, limited operational control<br />17<br />
    • 18.
    • 19. distributed computing paradigm<br />19<br />Map:Reduce + distributed storage:<br /><ul><li>Simplicity of online, stateless models
    • 20. Expressiveness of offline analysis
    • 21. Ease of management</li></li></ul><li>the map:reduce paradigm <br />Input data format is application-specific, specified by the user <br />Output is a set of &lt;key,value&gt; pairs <br />User expresses algorithm using two functions<br />Map is applied on the input data and produces a list of intermediate &lt;key,value&gt; pairs <br />Reduce is applied to all intermediate pairs with the same key. It typically performs some kind of merging operation and produces zero or more output pairs <br />Finally, output pairs are sorted by their key value<br />20<br />
    • 22. the map:reduce paradigm <br />21<br />Mapper<br />&lt;k1,v1&gt;<br />Mapper<br />&lt;k1,{v1,v3}&gt;<br />&lt;k2,v2&gt;<br />Reducer<br />&lt;k2,v2&gt;<br />&lt;k1,W1&gt;<br />Mapper<br />&lt;k1,v3&gt;<br />
    • 23. A SIMPLE MAP:REDUCE EXAMPLE<br />$ bin/hadoopdfs -cat /usr/joe/wordcount/input/file01 <br />Hello World Bye World <br />$ bin/hadoopdfs -cat /usr/joe/wordcount/input/file02 <br />Hello Hadoop Goodbye Hadoop<br />// Split up input files (MAP), iterate over chunks, reassemble results (REDUCE) <br />$ bin/hadoop jar /usr/joe/wordcount.jarorg.myorg.WordCount /usr/joe/wordcount/input /usr/joe/wordcount/output<br />$ bin/hadoopdfs -cat /usr/joe/wordcount/output/part-00000 <br />Bye 1 <br />Goodbye 1 <br />Hadoop 2 <br />Hello 2 <br />World 2 <br />22<br />
    • 24. a simple map:reduce example (bit.ly/bdyi0l)<br />18. public void map(LongWritable key, Text value, OutputCollector&lt;Text, IntWritable&gt; output, Reporter reporter) throws IOException {<br />19. String line = value.toString();<br />20. StringTokenizertokenizer = new StringTokenizer(line);<br />21. while (tokenizer.hasMoreTokens()) {<br />22. word.set(tokenizer.nextToken());<br />23. output.collect(word, one);<br />24. }<br />25. }<br />23<br />
    • 25. a simple map:reduce example (bit.ly/bdyi0l)<br />28. public static class Reduce extends MapReduceBase implements Reducer&lt;Text, IntWritable, Text, IntWritable&gt; {<br />29. public void reduce(Text key, Iterator&lt;IntWritable&gt; values, OutputCollector&lt;Text, IntWritable&gt; output, Reporter reporter) throws IOException {<br />30. int sum = 0;<br />31. while (values.hasNext()) {<br />32. sum += values.next().get();<br />33. }<br />34. output.collect(key, new IntWritable(sum));<br />24<br />
    • 26. Applications <br />&amp; <br />Outcomes<br />25<br />
    • 27. Lets REVIEW OUR DESIGN GOALs AGAIN<br />Classifiers are notorious for lack of explainability<br />Engineers and analysts needs to know what the classifier is missing<br />Engineers and analysts need to know about emerging threats<br />Analysts need “canned” reports along interesting dimensions<br />Machines need smart feature engineering<br />Develop a scalable system to provide deep insight into spammer campaigns<br />Double up as a platform for standard reporting<br />Also double up as a platform for adhoc analysis and data probing<br />Signal amplification and smart feature extraction platform<br />26<br />
    • 28. Our ANTISPAM ANALYTIC PLATFORM<br />Hadoop: Implements map reduce, written in Java but supports many other languages including Perl and C++ using the streaming interface<br />Feature engineering with small simple Perl programs for data extraction and transformation<br />SQL-like “Pig” programming language for data analysis and management<br />Mahout: data mining libraries that provide shrink- wrapped, scalable, sophisticated algorithms<br />Other proprietary algorithms and frameworks for specialized tasks<br />27<br />
    • 29. Various ASPECTS of A GRID DRIVEN SOLUTION<br />Standard reporting<br />Ad hoc querying<br />Campaign discovery from spam feedback using frequent item set mining<br />“Gaming” detection in notspam feedback using connected components<br />28<br />
    • 30. Top SPAMMY DOMAINS REPORT FOR 01/15/2010<br />29<br />key:noreply.amateurmatch.com|value:1164<br />key:goodmere.info|value:896<br />key:marketing.meredith.com|value:1078<br />key:verizon.net|value:822<br />key:reply.mb00.net|value:980<br />key:insideapple.apple.com|value:1094<br />key:facebookappmail.com|value:882<br />key:mydailymoment.com|value:849<br />key:thetwilightsaga.com|value:4671<br />key:adknowledgemailer6.com|value:859<br />key:freedollarspro.info|value:1164<br />key:smartreachmedia.com|value:1074<br />key:yahoo.es|value:877<br />key:ecomasher.com|value:1197<br />key:leasetrade-statusupdates.com|value:951<br />key:noreply.amateurmatch.comvalue:1164<br />
    • 31. AD HOC queries for ANTISPAM research<br />Identify domains that had few spam votes in the previous time window but have a high number of spam votes today<br />All IPs in the last hour that sent a particular URL pattern…or that sent any unknown URL &gt;500 times<br />Which domains/IPs suddenly increased their sending volume after a positive reputation change<br />Which FROM addresses exhibit low message size entropy<br />All messages that had nothing but a URL and the domain of the URL had low page rank<br />30<br />
    • 32. AD HOC QUERIES - Anatomy of a PIG QUERY<br />--- This includes some basic string functions, including splitting a string on the &apos;@&apos; character<br />register /homes/jpujara/pig_scripts/string.jar;<br />define splitEmail string.Tokenize(&apos;2&apos;,&apos;@&apos;);<br />--- Load up some data - incoming messages at a date and time, and our trusted user database<br />MESSAGES = load &apos;/projects/antispam/mta_feature_logs/$date*/*/*-$time*&apos; using com.yahoo.ymail.pigfunctions.AsStorage(&apos;__record_key__,firstrcpt,mailfrom&apos;) as (mid:chararray,to:chararray,from:chararray);<br />USERS = load &apos;/projects/antispam/TrustedUser.bz2&apos; using com.yahoo.ymail.pigfunctions.AsStorage(&apos;user,t&apos;) as (user:chararray,trusted:int);<br />--- Split the e-mail addresses into user+domain and generate the appropriate user-id for yahoo users and partners<br />EXPLODED_MESSAGES = FOREACH MESSAGES GENERATE to,FLATTEN(splitEmail(to)) as (user,udomain),FLATTEN(splitEmail(from)) as (sender,sdomain);<br />YAHOO_MESSAGES = FOREACH EXPLODED_MESSAGES GENERATE (udomain MATCHES &apos;.*yahoo.*&apos; ? user : to ) as yuser,sdomain;<br />31<br />--- Combine the message and sender domains with the trusted user data and select only trusted messages<br />YAHOO_MESSAGES_TRUST = JOIN YAHOO_MESSAGES by yuser, USERS by user;<br />TRUSTED_MESSAGES = FILTER YAHOO_MESSAGES_TRUST by trusted &gt; 0;<br />--- Group by domain, and generate a count, order by descending count<br />DOMAIN_GROUPS = GROUP TRUSTED_MESSAGES by sdomain;<br />DOMAIN_GROUPS_COUNT = FOREACH DOMAIN_GROUPS GENERATE group,COUNT(TRUSTED_MESSAGES) as count;<br />DOMAIN_GROUPS_ORDER = ORDER DOMAIN_GROUPS_COUNT by count DESC;<br />--- Output the results<br />STORE DOMAIN_GROUPS_ORDER into &apos;$targetdir/topDomains&apos;;<br />
    • 33. CAMPAIGN Discovery in SPAM Feedback<br />Frequent Itemset Mining<br />Classical method<br />Research interesting relationships between variables in a large database<br />Primarily applied for market basket analysis<br />Many good implementations<br />APRIORI<br />Easy to implement<br />Parallelizes moderately well but bottlenecks for extremely large data sets<br />Not very efficient with the number scans<br />ECLAT<br />Parallelizes easily <br />Amenable to a good grid implementation<br />Fewer scans of the dataset<br />Parallel FP GROWTH<br />Designed explicitly for systems like hadoop<br />Implemented in Mahout 0.2<br />32<br />
    • 34. Frequent item set – example dataset<br />33<br />
    • 35. Frequent ITEMSET MINING<br />34<br />Slide Courtsey: dortmund.de<br />
    • 36. Frequent itemset MINING on ONE DAY’s SPAM REPORTS<br />9 2595 (IPTYPE:none,FROMUSER:sales,SUBJ:It&apos;s Important You Know,FROMDOM:dappercom.info,URL:dappercom.info,ip_D:66.206.14.77,)<br />9 2457 (IPTYPE:none,FROMUSER:sales,SUBJ:Save On Costly Repairs,FROMDOM:aftermoon.info,URL:aftermoon.info,ip_D:66.206.14.78,)<br />9 2447 (IPTYPE:none,FROMUSER:sales,SUBJ:Car-Dealers-Compete-On-New-Vehicles,FROMDOM:sherge.info,URL:sherge.info,ip_D:66.206.25.227,)<br />9 2432 (IPTYPE:none,FROMUSER:sales,SUBJ:January 18th: CreditReport Update,FROMDOM:zaninte.info,URL:zaninte.info,ip_D:66.206.25.227,)<br />9 2376 (IPTYPE:none,FROMUSER:health,SUBJ:Finally. Coverage for the whole family,FROMDOM:fiatchimera.com,URL:articulatedispirit.com,ip_D:216.218.201.149,)<br />9 2184 (IPTYPE:none,FROMUSER:health,SUBJ:Finally. Coverage for the whole family,FROMDOM:fiatchimera.com,URL:stratagemnepheligenous.com,ip_D:216.218.201.149,) <br />9 1990 (IPTYPE:none,FROMUSER:sales,SUBJ:Closeout 2008-2009-2010 New Cars,FROMDOM:sastlg.info,URL:sastlg.info,ip_D:66.206.25.227,)<br />9 1899 (IPTYPE:none,FROMUSER:sales,FROMDOM:brunhil.info,SUBJ:700-CreditScore-What-Is-Yours?,URL:brunhil.info,ip_D:66.206.25.227,)<br />9 1743 (IPTYPE:none,FROMUSER:sales,SUBJ:Now exercise can be fun,FROMDOM:accordpac.info,URL:accordpac.info,ip_D:66.206.14.78,)<br />9 1706 (IPTYPE:none,FROMUSER:sales,SUBJ:Closeout 2008-2009-2010 New Cars,FROMDOM:rionel.info,URL:rionel.info,ip_D:66.206.25.227,)<br />9 1693 (IPTYPE:none,FROMUSER:sales,SUBJ:January 18th: CreditReport Update,FROMDOM:astroom.info,URL:astroom.info,ip_D:66.206.25.227,)<br />9 1689 (IPTYPE:none,FROMUSER:sales,SUBJ:eBay: Work@Home w/Solid-Income-Strategies,FROMDOM:stamine.info,URL:stamine.info,ip_D:66.165.232.203,)<br />35<br />2432 (IPTYPE:none,FROMUSER:sales,SUBJ:January 18th: CreditReportUpdate,FROMDOM:zaninte.info,URL:zaninte.info, ip_D:66.206.25.227,)<br />2447 (IPTYPE:none,FROMUSER:sales,SUBJ:Car-Dealers-Compete-On-New-Vehicles,FROMDOM:sherge.info,URL:sherge.info,<br />ip_D:66.206.25.227,)<br />
    • 37. Gaming DETECTION in NOTSPAM FEEDBACK<br /><ul><li>Spammers instrument accounts to vote “not spam” on emails that they send
    • 38. Delays classification of spamming IP addresses</li></ul>Throws off the classifiers if the feedback is not filtered well<br />Model the problem as a bipartite graph<br />Well known model for matching algorithms<br />Broadly applied in various fields like coding theory<br />A graph whose vertices are disjoint form disjoint sets U,V <br />There is an edge connecting every U to a vertex in V<br />36<br />
    • 39. Connected COMPONETS - EXPLAINED<br />Y1 = Yahoo user 1, Y2 = Yahoo user 2<br />IP1 = IP address of the host Y1 “voted” notspam from<br />37<br />y1<br />IP1<br />y1<br />SQUARING<br />weight = 2<br />y1<br />IP2<br />y1<br />
    • 40. Connected COMPONENTS for “GAMING” DETECTION<br />38<br />Set of IPs/YIDs used <br />exclusively for <br />voting notspam<br />Set of (likely new) <br />spamming IPs which <br />are “worth” voting for<br />y1<br />IP3<br />IP1<br />y2<br />IP4<br />IP2<br />y3<br />Set of <br />“voted on” IPs<br />Set of <br />“voted from” IPs<br />Set of Yahoo IDs<br />voting notspam<br />
    • 41. Connected Components - RESULTS<br />39<br />- Connnected components for IPsnotspam was voted from<br />
    • 42. Connected components - results<br />40<br />- Connnected components for IPsnotspam was voted on<br />
    • 43. CONCLUSIONS<br />We have had success leveraging parallel, stateful algorithms on grid systems to keep pace with polymorphic spam that evade traditional analysis and algorithms<br />Frequent Itemset Mining rapidly identifies cohesive campaigns in ISSPAM feedback<br />Connected Components amplifies weak signals in gamed NOTSPAM feedback and helps separate signal from noise in the feedback<br />Grid system based analysis platforms may be broadly applicable across the security domain<br />41<br />
    • 44. Apply Slide<br />Download Hadoop distribution<br />http://hadoop.apache.org<br />Try out Pig on standalone, single Linux box<br />Identify source data to aggregate<br />Start simple: IP patterns across web access logs<br />Begin with offline aggregation; yesterday’s attacks still interesting<br />Read Connected Components and Frequent Itemset Mining papers<br />Stop looking for a single, invariant “tell” – far too costly<br />Start thinking about co-occurrence of innocuous features <br />42<br />
    • 45. Resources for implementers<br />Hadoop setup, documentation and resources<br />http://hadoop.apache.org/<br />Pig documentation and resources<br />http://hadoop.apache.org/pig/<br />Mahout documentation and resources<br />http://lucene.apache.org/mahout/<br />Frequent itemset mining implementation repository<br />http://fimi.cs.helsinki.fi/src/<br />Connected components description<br />[link not yet live]<br />Ranger, Raghuraman, Penmetsa, Bradski, and Kozyrakis. Evaluating MapReduce for Multi-core and Multiprocessor Systems. In HPCA 2007<br />43<br />
    • 46.
    • 47. Connected COMPONENTS<br />45<br /><ul><li>Reg IP
    • 48. Cookie
    • 49. Username
    • 50. Birthday
    • 51. Reg IP
    • 52. Cookie
    • 53. Username
    • 54. Birthday
    • 55. Reg IP
    • 56. Cookie
    • 57. Username
    • 58. Birthday
    • 59. Reg IP
    • 60. Cookie
    • 61. Username
    • 62. Birthday
    • 63. Reg IP
    • 64. Cookie
    • 65. Username
    • 66. Birthday
    • 67. Reg IP
    • 68. Cookie
    • 69. Username
    • 70. Birthday
    • 71. Reg IP
    • 72. Cookie
    • 73. Username
    • 74. Birthday</li>

    ×