Abaca: The World's Most Effective Spam Filter


Published on


Published in: Technology, News & Politics
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • Unprecedented accuracy Accuracy monotonically increases as time and the number of protected users increases Easy to identify mistakes: Quarantine is ranked with rigorously calculated likelihood of the observation being correct so that virtually all mistakes are in the email “greymail” report Real-time rating: reaction times are nearly instantaneous Content independent: works on image spams, foreign spams, word salads, etc. 100% language-independent High performance Highly decisive; greymail list is typically <1% of all emails No training/ramp up period Immune to human rating errors User feedback is optional (personal preferences). No tuning (whitelists, blacklists, high-medium-low settings, etc)
  • Abaca: The World's Most Effective Spam Filter

    1. 1. Next Generation Protection Against a Dangerous New Generation of Spam Abaca The world’s most effective spam filter
    2. 2. Agenda <ul><li>Traditional spam defenses suffer </li></ul><ul><li>Abaca Corporation </li></ul><ul><li>Next-generation technology </li></ul><ul><li>Abaca Products </li></ul>
    3. 3. A History of Flawed Anti-spam Approaches Blacklists Slow and insufficient Punishes the innocent Rules-based Slow and labor intensive Content inspection Relies on end users reporting Consumes much memory Decoys and Honeypots Relies on slow humans to write rules, Trouble differentiating similar messages Collaborative checksum Reactive — damage already done Checksums can be defeated Greylisting Unacceptable delay easy for a spammer to defeat
    4. 4. Accuracy Claims have been Greatly Exaggerated Vendors trade false positives for accuracy
    5. 5. About Abaca <ul><li>Abaca: A Steve Kirsch Company </li></ul><ul><ul><li>Mouse Systems – 1 st Optical Mouse </li></ul></ul><ul><ul><li>Frame Technology – Desktop Publishing (Adobe) </li></ul></ul><ul><ul><li>Infoseek – Internet Search Engine (Disney) </li></ul></ul><ul><li>Founded during 2005 in San Jose, California </li></ul><ul><li>Next-generation anti-spam with unsurpassed accuracy </li></ul><ul><li>Battle-tested at Yahoo!Mail – a juicy spam target </li></ul><ul><li>Over 350 million email accounts under contract </li></ul>
    6. 6. Abaca Delivers Critical Component Of Messaging Security Effectiveness /Accuracy Ease of Administration Messaging Security : Large and Growing Market IDC: worldwide messaging security market 11.5% CAGR
    7. 7. Customer Successes <ul><ul><ul><li>“ Abaca offers superior e-mail security capabilities and has built a reputation for reducing unsolicited e-mail within mailboxes. We believe that by deploying Abaca’s solution with our anti-spam toolkit, we will offer Yahoo! Mail users not only added email security, but an enhanced user experience as well.” </li></ul></ul></ul><ul><ul><ul><li>John Kremer </li></ul></ul></ul><ul><ul><ul><li>Vice President of Yahoo! Mail </li></ul></ul></ul><ul><ul><ul><li>“ Their remarkable performance, combined with Abaca's scalability and flexibility guaranteed the performance we were looking for.” </li></ul></ul></ul><ul><ul><ul><li>Marco Schilling </li></ul></ul></ul><ul><ul><ul><li>Director of Technology, Terra Latin America </li></ul></ul></ul>
    8. 8. Next Generation Anti-spam Technology <ul><li>Real-time, crowd-sourced </li></ul>Abaca © 2010, All rights reserved
    9. 9. Abaca Works Because Spam Obeys Some Laws <ul><li>Defined by relationship between sender and receiver, </li></ul><ul><li>not the content </li></ul><ul><li>RCPT TO is the only familiar item </li></ul><ul><li>Must be sent in high volume </li></ul><ul><li>Spammers must send to people who collectively receive more spam than average </li></ul><ul><li>Recipient’s ham:spam mix is relatively constant </li></ul>Envelope Contains the Key not the message From: To:
    10. 10. Receiver Reputation: A Unique and Protected Algorithm <ul><li>Receiver reputation precisely differentiate spam from legitimate messages </li></ul><ul><li>Characterize (by passive observation) each protected user based on the % of spam they receive (receiver reputations) </li></ul><ul><li>Message ratings are based on each user's overall legitimate/spam ratio </li></ul><ul><li>Automatically learns and improves accuracy with real time legitimate/spam statistics for each protected user </li></ul>
    11. 11. 90% Spam 75% Spam 50% Spam 25% Spam 10% Spam Recipient Spam % = Receiver Reputation ReceiverNet ™ Receiver Reputation Legitimate Email Legitimate Email Spam Email Spam Email
    12. 12. 90% Spam 75% Spam 50% Spam 25% Spam 10% Spam Based on the Reputation of the Recipients this Message is Spam ReceiverNet ™ Receiver Reputation Legitimate Email Spam Email Spam Email Spam Email
    13. 13. 90% Spam 75% Spam 50% Spam 25% Spam 10% Spam Based on the Reputation of the Recipients this Message is Legitimate ReceiverNet ™ Receiver Reputation Legitimate Email Spam Email Legitimate Email Legitimate Email
    14. 14. ABACA PRODUCTS <ul><li>Designed for hosted mail providers </li></ul>Abaca © 2010, All rights reserved
    15. 15. CLX : Carrier Class Anti-spam Solution CLX Outbound Spam Filter CLX Spam Rating Engine CLX Solution CLX Spam Quarantine * Optionally deployed in-line as supplemental filter *
    16. 16. Simple Integration, Rapid Deployment <ul><li>Simple programming interface </li></ul><ul><li>Standard SMTP filtering protocols including milter </li></ul><ul><li>Seamless integration with messaging infrastructure </li></ul><ul><ul><li>Leverages existing feedback mechanisms </li></ul></ul><ul><ul><li>Tag, block, quarantine or deliver messages to easily conform to existing user behavior </li></ul></ul><ul><li>Self contained, no need to open ports to outside </li></ul>
    17. 17. Reduces Anti-spam Expenses <ul><li>Accurate, zero hour protection </li></ul><ul><li>Automatic real-time reputation updates </li></ul><ul><ul><ul><li>No waiting for new rules </li></ul></ul></ul><ul><ul><ul><li>No updating signatures </li></ul></ul></ul><ul><li>Reduces the burden on IT resources </li></ul><ul><li>Green requires one tenth the CPU cycles of conventional content-based filtering </li></ul>Abaca © 2010, All rights reserved
    18. 18. Abaca CLX Inbound Solution Components <ul><li>The Abaca CLX Anti-spam Solution includes the following components: </li></ul><ul><ul><li>abacam Client, available as a milter </li></ul></ul><ul><ul><li>(Note the abacam client is run on a standard MTA) </li></ul></ul><ul><ul><li>CLX Inbound Rating Server </li></ul></ul><ul><ul><li>CLX Feedback Server </li></ul></ul><ul><ul><li>CLX Admin </li></ul></ul><ul><ul><li>CLX Quarantine (optional) </li></ul></ul><ul><li>Individual hardware servers are not required for all components </li></ul><ul><li>The following slide illustrates the architecture of a typical inbound deployment </li></ul>
    19. 19. Scalable CLX Inbound Architecture CLX Server CLX Server Internet Quarantine Quarantine Email Server Email Server CLX Server Email Server Message rating CLX Admin Inbound Outbound MTA abacam MTA abacam MTA abacam SPAM
    20. 20. CLX Outbound Solution Components <ul><li>Abacam Client, available as a milter for </li></ul><ul><ul><li>Milter plugs into SendMail, PostFix and Zimbra </li></ul></ul><ul><li>CLX Outbound Rating Server </li></ul><ul><li>CLX Administrator Console </li></ul><ul><li>CLX Outbound Quarantine </li></ul><ul><li>Components can share hardware servers </li></ul>* Note the abacam client is run on a standard MTA)
    21. 21. Outbound Spam Filter Deployment CLX Outbound Quarantine Email Server Email Server CLX Outbound Server Inbound training information Email Server Admin Rate request Outbound CLX Outbound Server CLX Outbound Server CLX Outbound Quarantine Rate response Feedback Outbound Outbound MTA abacam MTA abacam MTA abacam MTA abacam SPAM
    22. 22. Abaca CLX Clustering <ul><li>Redundant component deployment </li></ul><ul><ul><li>On the MTAs </li></ul></ul><ul><ul><li>Abacam requests ratings from multiple severs </li></ul></ul><ul><ul><li>Reputation data is synchronized and shared </li></ul></ul><ul><li>If CLX Server hardware fails </li></ul><ul><li>Abacam routes requests to the remaining servers </li></ul><ul><ul><li>User preferences are preserved </li></ul></ul><ul><ul><li>Message flow is not interrupted </li></ul></ul><ul><ul><li>Accuracy is maintained </li></ul></ul><ul><li>When the hardware is serviced and is back online </li></ul><ul><ul><li>CLX Server will be included in rating requests </li></ul></ul>
    23. 23. Abaca CLX Cluster Architecture CLX Rating Cluster Client (abacam) … Client (abacam) Client (abacam) <ul><li>Incoming Request </li></ul><ul><li>Rate Inbound </li></ul><ul><li>Rate Outbound </li></ul><ul><li>Feedback </li></ul>Load Balancer (abacam) CLX Rating Server CLX Rating Server <ul><li>Synchronization </li></ul><ul><li>Outbound Messages </li></ul><ul><li>Feedback </li></ul>Cluster Exploded View > 50 million mailboxes - Mail flow partitioned between CLX servers < 50 million mailboxes – Round-robin partitioning used Blue fill indicates Abaca components
    24. 24. Abaca CLX Multi-site Support <ul><li>Redundancy at the site level is supported though synchronization between each site </li></ul><ul><ul><li>Servers synchronized and share reputation data </li></ul></ul><ul><li>Synchronization does not require significant bandwidth between servers or sites </li></ul><ul><li>Emails for a user can come into either site at any time and will be processed with the same results </li></ul><ul><li>When a site becomes unavailable, the remaining site will rate the entire message flow </li></ul><ul><ul><li>User preferences are preserved </li></ul></ul><ul><ul><li>Message flow is not interrupted </li></ul></ul><ul><ul><li>Accuracy is maintained </li></ul></ul>
    25. 25. Abaca Multi-site CLX Deployment CLX Server CLX Server CLX Server CLX Server Feedback and outbound synchronization Failover Rate requests Rate requests
    26. 26. Abaca Difference <ul><li>Proactive Real-Time Defense </li></ul><ul><li>Learns in real time so not playing catch up </li></ul><ul><li>No Blacklists to update </li></ul><ul><li>No content rules to maintain </li></ul><ul><li>Set and forget </li></ul><ul><li>Language-independent </li></ul><ul><li>Encryption-independent </li></ul><ul><li>Rated quarantine highlights useful mail </li></ul>
    27. 27. For more information Abaca © 2010, All rights reserved <ul><li>John Jefferies </li></ul><ul><li>General Manager and CMO </li></ul><ul><li>[email_address] </li></ul><ul><li>408.205.5320 </li></ul>
    28. 28. ADDITIONAL SLIDES Abaca © 2010, All rights reserved
    29. 29. Attack Prevention Technique <ul><li>They could buy a box and pollute one global server </li></ul><ul><ul><li>Limit the # of daily contributions per unique user </li></ul></ul><ul><li>They could use accounts within an ISP to create good reputations at that ISP to allow spammers to attack that ISP from the outside </li></ul><ul><ul><li>Yahoo could Use someone else’s global </li></ul></ul><ul><li>Spammers buying good lists and using them exclusively </li></ul><ul><ul><li>Make any unsolicited emails being bad, regardless of reputation of receiver. So anyone sending out mass mailings all of a sudden gets caught, regardless of list quality </li></ul></ul><ul><ul><li>Use logistic regression filter </li></ul></ul><ul><ul><li>If spammers are using fixed IP range, block those </li></ul></ul><ul><ul><li>Good list will eventually become bad (the weaker addresses) </li></ul></ul>
    30. 30. When it Comes to Filtering Accuracy Matters * For a 1,000 person entity whose employees receive 50 messages / day
    31. 31. >100 times faster than other approaches <ul><li>The core algorithm is very simple </li></ul><ul><ul><li>currently rates at over 100,000 msgs/sec/core </li></ul></ul><ul><li>A single quad core PC w/32G RAM can easily handle the ratings for 20M users with just 5% of the CPU capacity </li></ul><ul><ul><li>Cost is about $4,000 </li></ul></ul>
    32. 32. Accurate rating allows a personalized greyzone <ul><li>One size doesn’t fit all: some hate FPs; others hate spam </li></ul><ul><li>Each user controls the level of acceptable false positive and false negative rates </li></ul><ul><li>At low user count, the “review zone” is less than 1 message out of 100 </li></ul>
    33. 33. Accuracy increases over time <ul><li>Each message is rated using the real-time data </li></ul><ul><ul><li>No time delay for push “updates” </li></ul></ul><ul><li>Huge number of raters </li></ul><ul><ul><li>Every rcpto is a rater </li></ul></ul><ul><li>Every rater is extremely accurate </li></ul><ul><ul><li>It is the user’s statistics that we use, not their human opinion (which is far less accurate than their stats due to phish, operator error, etc) </li></ul></ul><ul><li>The raters work for us 24x7 (for free) </li></ul><ul><ul><li>Other systems rely on human feedback which is sporadically available </li></ul></ul><ul><li>Our raters “decide” instantly </li></ul><ul><ul><li>No time delay; messages are rated before the content is received </li></ul></ul>
    34. 34. Who Uses Abaca? Market Solution Product Enterprise CLX Hosted Solution CLX Solution CLX Outbound Cloud Service (beta) xSPs Complete Solution Abaca Rating Engine Outbound Spam Filter CLX Solution CLX Engine Only CLX Outbound VMware Customers Virtual Appliance VPG SMBs Hardware Appliance EPG Education Hardware Appliance Virtual Appliance Outbound Spam Filter EPG VPG Outbound Filter