• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Yahoo! Mail antispam - Bay area Hadoop user group

Yahoo! Mail antispam - Bay area Hadoop user group






Total Views
Views on SlideShare
Embed Views



23 Embeds 1,727

http://developer.yahoo.net 602
http://www.readwriteweb.com 299
http://developer.yahoo.com 205
http://www.hadoopsphere.com 196
http://www.slideshare.net 156
http://readwriteweb.com.br 106
http://webholic.com.br 41
http://linkhay.com 36
https://developer.yahoo.com 24
http://feeds.feedburner.com 23
http://readwrite.com 18
http://static.slidesharecdn.com 10
http://2868824907842590784_18d887540e821527539eb43f8f9aa97f67770712.blogspot.com 1
http://webcache.googleusercontent.com 1
http://rss2.com 1
http://www.cnliam.com 1
http://www.abhinavnews.info 1
https://readitlaterlist.com 1
http://eberbach.pl 1
http://busk.com 1 1
http://www.hanrss.com 1
http://www.tuicool.com 1



Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
Post Comment
Edit your comment
  • Who knows what Yokai are? Shape-shifters from Japanese mythology. Many other examples, e.g. Proteus, who would tell you the future, but first you had to capture him. Just like the gods, Change shape to avoid capture* vary over IP, vary over content, vary over template features (e.g. document structure, subjects, size entropy)
  • In abuse, these are “shape shifters.”They vary many aspects of the message to avoid detection: IPSubjectContentFor example, these four messages are obviously built from a single template, but changing its shape to avoid capture. How to catch?In the past: + Heuristics & Regex + Dictionary (URLdb) + Invariant metadataChallenges: + slow to write+ difficult to write+ easy to evade
  • Here is a third type of shape-shifting spamFor all of these: attackers have distinct advantage, because they can change most aspects and still get through
  • 1.3 sextillion (1.3e21) variations, almost all can be recognized by human being in milisecondsspammers learned they can change any variable to hide from bulk filtershttp://cockeyed.com/lessons/viagra/viagra.html
  • These bastards… the most despised doctors on the InternetAlmost all pages resolve through numerous HTML/Javascript redirectors to this page
  • Daniel Geer said, there are targets of CHANCE and targets of CHOICE. Small businesses are in the former camp, catching the miscellaneous attacks out there.Increasingly, larger companies are TARGETS OF CHOICE, meaning the bad guys a) specifically tailor their attacks based on known vulnerabilities, and b) use feedback loops to improve the effectiveness of them.
  • This is what a targetted attack profile looks like: After you patch, they almost stop trying
  • One example of such a clearly targeted attack: 400KB of style gibberish embedded in a style sheet, completely throws out our parsersMaybe ASCII art spam, or something else that couldn’t be caught by simple pattern matchingThis is what our filters see: a stream of ASCII that is deliberately using multiple layerse.g. here, a TinyURL redirector, further obfuscated with non-printing HTML, spaces, and CSS chaffTo fight in olden days, hand-written regex to identify a patternOR heuristic on some invariant part of the message. But what is invariant? dozens of TinyURL clonesdozens of HTML and CSS tricks2^32 IP addressesinfinite FROM addressesinfinite SUBJECT lines…
  • Sent by botnetsThis is Reactor Mailer; controlled Srizbi from the McColo datacenters until Nov 2008This is the template for Stormbot; notice it has control variables for all the settingsWhile most of these came in through SMTP port 25, now they are increasingly hitting HTTP and port 80
  • Historically, POINT SOLUTIONS address each problem individuallyregexheuristicWouldn’t this be better if this guy could use more than one finger at a time?Something is *almost over the limit* along one dimension and *almost over the limit* along another.Message from IP that sends 80% good mail, with tinyurl that we don’t recognize, that was addressed to 40 people.*PRIOR PROBABILITY**COMPOSITE SCORE*
  • Scale forces simplistic architectures; Feedback based architectures always lag behind the spam campaignFeedback also has many segments;- Personal preference spam: “I didn’t like this week’s Amazon gold box deals but I liked last week’s messages from Amazon”- Annoyance emails from legitimate bulk mailers: “This coupon is coming far too often these days”-Listserver spam: “This finance group - Newsletter messages that are no longer interesting to the user: “Gosh I am so not into that band any more”sometimes sends me stock spam”Traffic to a small enterprise domains can be restricted with firewall rules etc but large free mail provider traffic is full of corner casesCompounding the problem is the fact that adoption of DKIM and SPF has been slow, especially internationally and in emerging economies.But make no mistake, some of these spammers are very cleverIts more fruitful to target yahoo or google than to build a generic spam engine
  • Lets looks at what is in place right now in terms of an architecture; Most large scale systems have some components from gen1 technologiesProvide attack mitigation and operational flexibility, highly explainable. Not durable, expensive to keep pace with fast morphing spamProprietary implementations, not very scalable, steep learning curveReactive and usually late
  • Two ways this has been solved in the past: Machine management…Both systems, because of scale, were limited to looking at small pieces of data – an IP, a URL, etc.
  • In this talk we’ll introduce Hadoop, an open-source grid computing environment with applications to fighting abuse. We’ll talk about how Hadoop can be applied to polymorphic spam and abuseAbout three years ago, Doug Cutting released version 0.15 of Hadoop, an open-source platform inspired by Google’s proprietary Map:Reduce algorithm“Supercomputer” – petabytes of storage, terabytes of RAM allow “needle in the haystack” even at Y!Mail scalehundred of featureshundreds of billions of recordstrends buried in global data
  • Hadoop is the most prevalentAlso “Ngrid” and “Sun’s GridEngine” are other alternatives
  • Input data format is application-specific, specified by the user Output is a set of pairs User expresses algorithm using two functionsMap is applied on the input data and produces a list of intermediate pairs Reduce is applied to all intermediate pairs with the same key. It typically performs some kind of merging operation and produces zero or more output pairs Finally, output pairs are sorted by their key value
  • Toy exampleProvide some insight into what a map reduce program looks like, looks very much like unix command line
  • Java code to highlight the mapper, mapper simply adds each word to a set and emits a count of 1 for each time the word is seen
  • The reducer simply sums the values for each word, draw attention to line 32While this is a toy example, it should give a fair idea about how to structure a problem to be solvable by map reduce. The key takeaway is that writing even native map reduce programs can be quite simple and executing it even simpler
  • Take the audience progressively through more and more sophisticated applications, starting from basic reporting and ending in outbound spammer analysis based on SWARM features
  • Knowing the accurancy of your SVM/Bayes classifier puts you in no better situation to ask and answer what type of spam is leaking; and we know spammers are constantly probing80% of the spam/content classification problem is in smart feature engineering
  • Lets looks at how our/Yahoo’s platform looks like Perl programs for feature engineering make it very easy and flexibleHadoop with its pig support is already well suited as a platform for adhoc data analysisFor deep data mining, open source mahout
  • We will look at the hadoop is four different settings;
  • * In antispam, these basic reports combined with human review form a barrier against highly directed attacks that exploit system weaknesses* Note how easy it is to slice and dice your data and write fairly sophisticated reports using pig/streaming. It is critical in antispam systems that the reporting platform be flexible and provide a lot of expressive power, hadoop and pig achieve that.*
  • Previous such queries were against small samples, now we can do it against the full data set and get highly accurate results in a very short amount of timeAlternate architectures such as OLAP are too expensive at this scale
  • * Pig is a data flow specification language. Its like SQL but unlike SQL it is better suited for data flow control. * In antispam, these basic reports combined with human review form a barrier against highly directed attacks that exploit system weaknessesNote how easy it is to slice and dice your data and write fairly sophisticated reports using pig. It is critical in antispam systems that the reporting platform be flexible and provide a lot of expressive power, hadoop and pig achieve that.*
  • -- People who bought eggs also bought bread
  • * We ran frequent itemset on one day’s spam votes, the results are striking.* Notice in the above example how the same campaign [the same FROMUSER] is being managed with different templates for subjects and URLs and is also originating from different IPs* Others records in the background are the result of the freq itemset mining algorithm as well and map very closely with spam campaigns.
  • Develop a bipartite graph of users and the IPs they vote fromSquaring of the graph give rise to connected componentWeight of the connected component is a measured by the number of vertices that share the component.
  • GamingIPs are IPs that the spammers try to whitelist in advanceDetected them by extending the connected component view on Ips the notspam is voted on
  • The results are quite spectacular!! There is a massive amount of “gaming” going on with “notspam feedback” and there are only a handful of Ips that are doing this. There are a large number of smaller components not shown in the results above
  • The results are less stronger – notice the two smaller weaker clusters in row 3 and 4The big takeway is that such unsupervised matching algorithms are going to be extremely power amplifiers of signals and can be used to rapidly separate out noise from signal.Imagine this being applied on traffic with more items such as IPs, message subjects, size of messages, fuzzy signatures etc.
  • We encourage and invite others to try hadoop in anti spam and anti abuse architectures and share their experiences with us.
  • Three users known badsame IP leads to new cookiesame cookie leads to new birthdayetc.*AMPLIFICATION OF SMALL SIGNAL*

Yahoo! Mail antispam - Bay area Hadoop user group Yahoo! Mail antispam - Bay area Hadoop user group Presentation Transcript