Winning the Big Data SPAM Challenge__HadoopSummit2010
•
1 like•575 views
This document discusses using Hadoop to process large amounts of spam data. It describes different types of spam, including email spam, social media spam, and web spam. It then explains why Hadoop is well-suited for spam processing due to its ability to parallelize tasks and handle large datasets. Sample system architectures and heuristics for spam detection are presented, such as analyzing IP addresses, link patterns, and content. Metrics like Jaccard similarity and arrival times can also help evaluate spam. Overall, the document advocates using Hadoop to gain insights from massive spam datasets through simple solutions that can effectively capture the majority of spam.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Winning the Big Data SPAM Challenge__HadoopSummit2010
Similar to Winning the Big Data SPAM Challenge__HadoopSummit2010 (20)
More from Yahoo Developer Network
More from Yahoo Developer Network (20)
Recently uploaded
Recently uploaded (20)
Winning the Big Data SPAM Challenge__HadoopSummit2010
- 1. Winning the Big Data Spam Challenge Erich Nachbar Stefan Groschupf Florian Leibert
- 2. Spam Types - Email Spam • What do spammers do? • Many domains • Cycle through IPs (TOR, bulk blocks) • Bulk account creation (increase IP reputation) • Break captchas (Mechanical Turk) • Common names (e.g. http://www.census.gov/genealogy/names/dist.male.first)
- 3. Spam Types - Social Media • Spam Carriers • Blog Postings • Comments • Friend Requests • ... • Spam Generation through • Actual User Accounts (Hacked / User Virus) • Bot Accounts
- 4. Spam Types - Social Media • Differences • Detection is the same • Account treatment is different (cancel vs. clean) • 99% of all Spam contains URLs: • Ignore text-only messages. • Look at the URL not the text.
- 5. Spam Types - Web Spam • Goal: influence search engine results • Link farms • Keyword, Meta tag stuffing • Hidden or invisible unrelated text • Scraper sites • Spam blogs
- 6. Why process Spam in Hadoop? • Easy to parallelize • Bucketization • User • Date • Source • etc. • Count models (probabilities) are very "hadoopy"
- 7. Why process Spam in Hadoop? • Large data sets • More samples ~ better results • Algorithms require preprocessing • Existing code • e.g. url-parsers, bayes implementations •
- 9. Heuristics for Spam Detection •Easy to compute, Group-By & Count •Captcha solving rates •Source IP/Email • Historic vs. current volume • Reputation • Link • Rrequency, position, ratio
- 10. Heuristics for Spam Detection • Content • Self similarity • Hash of media content • Keywords
- 11. Looking at arrival times • Inter-arrival times • Fast Fourier Transform • Timestamps -> frequency space
- 12. Evaluating content • Jaccard similarity • Bucketize by user / source email • s1 = S(x1,x3,x5), s2 = S(x2,x4,x6) • Easy with Hadoop • map: emit user_id (K), text (V) • reduce: • Even simpler: • # links / user • # complaints, spam tags / user • etc.
- 13. Solutions - Pay Level Domain • Requires payment at Top Level Domain • Simple heuristic • Much simpler than Trust-Rank, Page-Rank, etc. Hsin-Tsang Lee, Derek Leonard, Xiaoming Wang, and Dmitri Loguinov IRLbot: Scaling to 6 Billion Pages and Beyond
- 14. Demo
- 15. Take Aways • Spam Reports are important • Rolling real-time Ham & Spam Samples • Have Knobs to turn (e.g. over JMX) • Simple solutions can get you pretty far, the easy 80% • Spammers adapt very fast, stay agile • Try to break your own system
- 16. Thank you! erich@quantifind.com, @enachb sg@datameer.com, @datameer flo@leibert.de, @floleibert