ASSP: Extracting the Ham from Spam -- by David J. Young

  • 6,300 views
Uploaded on

http://www.uniforum.chi.il.us/slides/assp.ppt

http://www.uniforum.chi.il.us/slides/assp.ppt

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
  • slide 11 mixed up false positive and false negative , interchange them
    Are you sure you want to
    Your message goes here
  • Thank you! This is a very good presentation! This would have helped me alot when I started using ASSP about 6-7 years ago.

    Should post original presentation on ASSP wiki.
    Are you sure you want to
    Your message goes here
  • great presentation of history of spam.
    Thanks
    If you like to know more about scammers information please visit my blog http://scambaitings.blogspot.com/
    Are you sure you want to
    Your message goes here
No Downloads

Views

Total Views
6,300
On Slideshare
0
From Embeds
0
Number of Embeds
1

Actions

Shares
Downloads
0
Comments
3
Likes
2

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Extracting the Ham from Spam David J. Young
  • 2. Introduction
    • History
    • Spam
    • Terminology
    • ASSP
    • Benchmarks
    • Demo
    • Questions
  • 3. History
    • Where did the term “spam” come from?
  • 4. SP iced h AM
  • 5. SPAM sketch http://www.youtube.com/results?search_query=spam+monty+python http://video.google.com/videosearch?q=spam+monty+python
    • Scene:   A cafe.  One table is occupied by a group of Vikings wearing horned helmets.  Whenever the word "spam" is repeated, they begin singing and/or chanting.  A man and his wife enter.  The man is played by Eric Idle, the wife is played by Graham Chapman (in drag), and the waitress is played by Terry Jones, also in drag.
    • Man: You sit here, dear. Wife: All right. Man: Morning! Waitress: Morning! Man: Well, what've you got? Waitress: Well, there's egg and bacon; egg sausage and bacon; egg and spam; egg bacon and spam; egg bacon sausage and spam; spam bacon sausage and spam; spam egg spam spam bacon and spam; spam sausage spam spam bacon spam tomato and spam; Vikings: Spam spam spam spam... Waitress: ...spam spam spam egg and spam; spam spam spam spam spam spam baked beans spam spam spam... Vikings: Spam! Lovely spam! Lovely spam! Waitress: ...or Lobster Thermidor a Crevette with a mornay sauce served in a Provencale manner with shallots and aubergines garnished with truffle pate, brandy and with a fried egg on top and spam. Wife: Have you got anything without spam? Waitress: Well, there's spam egg sausage and spam, that's not got much spam in it. Wife: I don't want ANY spam! Man: Why can't she have egg bacon spam and sausage? Wife: THAT'S got spam in it! Man: Hasn't got as much spam in it as spam egg sausage and spam, has it? Vikings: Spam spam spam spam... (Crescendo through next few lines...) Wife: Could you do the egg bacon spam and sausage without the spam then? Waitress: Urgghh! Wife: What do you mean 'Urgghh'? I don't like spam! Vikings: Lovely spam! Wonderful spam! Waitress: Shut up! Vikings: Lovely spam! Wonderful spam! Waitress: Shut up! (Vikings stop) Bloody Vikings! You can't have egg bacon spam and sausage without the spam. Wife: I don't like spam! Man: Sshh, dear, don't cause a fuss. I'll have your spam. I love it. I'm having spam spam spam spam spam spam spam beaked beans spam spam spam and spam! Vikings: Spam spam spam spam. Lovely spam! Wonderful spam! Waitress: Shut up!! Baked beans are off. Man: Well could I have her spam instead of the baked beans then? Waitress: You mean spam spam spam spam spam spam... (but it is too late and the Vikings drown her words) Vikings: Spam spam spam spam. Lovely spam! Wonderful spam! Spam spa-a-a-a-a-am spam spa-a-a-a-a-am spam. Lovely spam! Lovely spam! Lovely spam! Lovely spam! Lovely spam! Spam spam spam spam!
  • 6. Spam Spam Spam lyrics
    • Lovely spam, wonderful spa-a-m, Lovely spam, wonderful S Spam, Spa-a-a-a-a-a-a-am, Spa-a-a-a-a-a-a-am, SPA-A-A-A-A-A-A-AM, SPA-A-A-A-A-A-A-AM, LOVELY SPAM, LOVELY SPAM, LOVELY SPAM, LOVELY SPAM, LOVELY SPA-A-A-A-AM... SPA-AM, SPA-AM, SPA-AM, SPA-A-A-AM!
  • 7. What is spam?
    • Unsolicited Bulk e-mail (UBE)
    • Unsolicited Commerical Email (UCE)
    • “ The abuse of electronic messaging systems to send unsolicited, undesired bulk messages”
  • 8. The cost of spam
    • Productivity – It is estimated that 80-85% of all email is spam
    • Payload may contain malware (virus, worm, trojan, etc.)
    • Internet bandwidth
  • 9. How do spammers get e-mail addresses?
    • Replying to a spam e-mail
    • Auto-responders (vacation)
    • Viewing HTML spam (web beacons)
    • Clicking on URLs to websites listed in spam
    • Chain e-mail (MUA virus)
    • Mining
      • Usenet postings/message boards/chat rooms
      • Usenet article message-IDs
      • Company or personal websites
      • DNS SOA records
      • whois database
    • Opt-out websites
    • E-mail worms harvesting address books
    • Shady businesses selling addresses to spammers
    • Dictionary attacks
    • Zombies
  • 10. Anti-spam best practices
    • Turn off email “preview”
    • Use throw away email addresses
    • Do not use an auto responder
    • Do not read spam
    • Do not click on URLs in spam
    • Give your e-mail address only to closely trusted acquaintances
    • Use images or other obfuscation techniques
    • Googling for your email address
    • Use a good spam filter
  • 11. Terminology True Positive (*****SPAM*****) False Positive SPAM (Positive) False Negative (*****SPAM*****) True Negative Not SPAM (Negative) Identified as SPAM Not Identified as SPAM
  • 12. xxxxx Listing
    • Whitelisting
      • A list of email addresses which would generally never send you spam
    • Blacklisting
      • A list of email addresses or domains you do not wish to receive any email from
    • Greylisting
      • Temporarily reject an unknown email by imposing a fixed delay before accepting email (ASSP calls this Delaying due to a name conflict)
    • Redlisting
      • Keeps an address off the whitelist
  • 13. More ASSP terms
    • Spam Lover
    • Spam Bucket
    • Honeypot
    • Postmaster
    • Bayesian
    • MTA
    • MUA
    • SMTP
  • 14. Processing matrix No processing (also doesn’t contribute to spam/nospam collections) Redlist (but does contribute to spam/nospam collections) Doesn’t contribute to whitelist Spam Lover Normal ASSP operation Contributes to whitelist Unfiltered Mail Filtered Mail
  • 15. What is ASSP?
    • A nti- S pam S MTP P roxy
      • “ An Open Source platform-independent transparent SMTP proxy server that leverages numerous methodologies and technologies to both rigidly and adaptively identify spam.”
      • -- wikipedia.org
  • 16. Theory of Operation
    • When you install ASSP a colony of super-intelligent thermophilus bacteria takes up residence on your CPU and begin reading all your email. They communicate using radio waves directly with the CPU and interface with the ASSP software choosing between spam and nonspam mail.
    • If you choose to read further this myth will be sadly dispelled, and I take no responsibility for the consequences.
    • However, you can always refer your users to this slide to prove to them that their email is actually being filtered by super-intelligent bacteria.
  • 17. True Theory of Operation
    • ASSP uses three complementary strategies to allow good email and to block unsolicited email
      • Whitelisting
      • Spambuckets
      • Bayesian filtering
    • Local mail domain users are not whitelisted
  • 18. ASSP Implementation
    • Version 1.2.5
    • It is a single Perl script
    • 360 KB
    • 10,000 lines
    • Built in web server
    • Built in Pseudo-SMTP server
  • 19. ASSP Target User Base
    • ASSP’s primary target audience is mail administrators or system administrators at smallish institutions. If you operate an ISP or a mailhost with a heterogeneous user base, you may not have a good enough consensus about what is considered spam or is not. It should work well with between 1 and 300 client addresses and a mail volume of up to around 100,000 messages per day. Testing has not been done to verify these ranges
    • ASSP is not for the following:
      • Individual clients -- ASSP must be installed together with a SMTP server
      • Domains which receive mail indirectly, for example if you use fetchmail
  • 20. ASSP Philosophy
    • Reject SPAM before the SMTP server
    • Work with any SMTP MTA
    • Adapt quickly as spammers change attack strategies
    • Require low maintenance after initial setup
  • 21. Main ASSP capabilities
    • Automatic Whitelisting
    • Spam Traps
    • Bayesian filtering
    • Greylist
    • Whitelist RE Matching
    • Email interface
    • Mail Analyzer
    • Automatic Statistics
    • SPF (Sender Policy Framework)
    • DNSBL (DNS Black Lists)
    • ClamAV virus scanner
    • Mail host Headers
  • 22. ASSP Features
    • Uses existing MTA and MUA’s
    • Runs on Linux, Unix, Windows, OS X, and more
    • Automatic whitelist – no-one you email will ever be blocked
    • Redlist keeps an address off the whitelist
    • Uses honeypot type spambucket addresses to automatically recognize spam and update your spam database
    • Bayesian filter intelligently classifies email into spam and non-spam
    • Supports site-defined regular expressions to identify spam or non-spam email
    • Accepts whitelist submissions and spam error reports by authorized email
    • Browser based setup
    • Keeps spam statistics for your site
    • Recognizes Mime encoded and other camouflaged spam
    • Can listen on more than one smtp port
    • Basic anti-virus filtering using the ClamAV virus databases
    • Optionally blocks no mail but adds an email header and/or updates the message subject (*****SPAM*****)
    • Can block spam-bombs (when spammers forge your domain in the from field)
    • More
  • 23. ASSP Flexibility
    • Whitelist-only mode
    • Don’t filter, just tag subject line
    • Let specific addresses receive SPAM
    • Use a mail list behind ASSP
    • Use ASSP with redundant MX domains
    • Web based configuration
  • 24. ASSP Mail Processing
    • What order does ASSP process mail to check if it is spam?
      • Local or whitelisted?
      • Blacklisted Domain?
      • Spam Helo?
      • Addressed to spam-bucket?
      • Mail bomb?
      • Blocked attachment?
      • Matches expression to identify non-spam?
      • Matches expression to identify spam?
      • Bayesian evaluation
    • If the message is identified as spam at any step along the way it goes to the spam directory. If the message is local or whitelisted it goes to the notspam directory.
  • 25. Installation Overview
    • Install ASSP and dependencies
    • Configure ASSP
    • Put ASSP in test mode
    • Modify mail flow of test user(s)
    • Test that it is working
    • Prime the system
    • Create the Bayesian database
    • Automate daily Bayesian database updates
    • Monitor spam filtering
    • Correct false negatives and false positives
    • Take ASSP out of test mode
    • Train user community
    • Modify mail flow of trained users
  • 26. ASSP Installation
    • Install Perl
    • Install Perl modules from CPAN
      • Compress::Zlib NEEDED - Standard Perl installation
      • Digest::MD5 NEEDED - Standard Perl installation
      • Time::HiRes NEEDED - Standard Perl installation
      • Net::DNS NEEDED TO RUN RBL, SPF and 1.2.X
      • Email::Valid OPTIONAL, BUT ADVISED
      • File::ReadBackwards OPTIONAL, BUT ADVISED
      • Mail::SPF::Query OPTIONAL
      • Mail::SRS OPTIONAL
      • Sys::Syslog OPTIONAL
      • Net::LDAP OPTIONAL :: NEEDED IF YOU RUN LDAP
      • Win32::Daemon NEEDED to run as a service on Windows
    • No installation script
      • GUNZIP assp.tar.gz to /usr/local/assp
      • In /usr/local create the following directories:
        • assp/spam
        • assp/notspam
        • assp/errors
        • assp/errors/spam
        • assp/errors/notspam
  • 27. Configure ASSP
    • Start ASSP
      • perl assp.pl
    • Configure ASSP
      • http://127.0.0.1:55555
      • Login: <empty>
      • Password: nospam4me (default)
    • Beware of the “Show Advanced Configuration” Option
  • 28. ASSP Configuration
  • 29. Initial Configuration
    • Change values for
      • “Web Admin Password”
      • “Accept All Mail”
      • “Local Domains”
      • “Spam Error”
      • “Spam Addresses”
        • Addresses of recipients at your site that only receive spam (website spam-bait, ex-employees)
  • 30. Mail Flow Internet Mail Svr Clients Inbound Outbound Internet ASSP Mail Svr Clients Inbound Outbound Internet ASSP Mail Svr Clients Internet Mail Svr Clients with ASSP Internet Mail Svr ASSP Clients Invalid
  • 31. Email Flow Internet ASSP GroupWise/ Exchange Clients Inbound Outbound MTA Internet GroupWise/ Exchange Clients MTA ASSP ASSP MTA smtp0 in out spam Not spam white red black grey Bayesian DB Errors 125 25
  • 32. 1999 This is an email that is being sent to the Internet. Th This is an email that is GWIA MTA POA GroupWise Internet
  • 33. 2003 Internet GWIA MTA POA GroupWise sendmail Virtuser table aliases Internet MTA DNS Block List
  • 34. 2004 GWIA MTA POA GroupWise sendmail Virtuser table aliases Internet MTA sendmail SpamAssassin SpamAssassin Internet
  • 35. 2006 GWIA MTA POA GroupWise sendmail Virtuser table aliases Internet MTA ASSP sendmail ASSP spam Not spam white red black grey Bayesian DB Errors sendmail SpamAssassin SpamAssassin Internet
  • 36. Phase In GWIA MTA POA GroupWise sendmail Virtuser table aliases Internet MTA ASSP sendmail ASSP spam Not spam white red black grey Bayesian DB Errors sendmail SpamAssassin SpamAssassin Internet
  • 37. Flow with Anti-Virus Internet ASSP Mail Svr Clients Internet ASSP Clients Inbound Outbound Antivirus Mail Svr Antivirus
  • 38. Flow with Groupware
    • To use ASSP with Exchange, Lotus Notes or GroupWise, you’ll also need to implement a “smarthost” relay like sendmail, qmail, postfix, exim or one in a number of others
    Internet ASSP Groupware Clients Inbound Outbound MTA Internet Groupware Clients MTA ASSP
  • 39. DNSBL vs Greylist
    • The ASSP Greylist supercedes DNSBL
    • ASSP “Greylist” is not to be confused with “Greylisting”
    • Use of DNSBL is discouraged (If a DNSBL lookup blocks, ASSP will block due to it’s multiplex design)
  • 40. Penalty Box
    • This will blacklist an SMTP server for about 72 hours or so from sending to your server if they violate basic SMTP connection conventions over a certain threshold.
  • 41. SMTP Ports
    • For example, internet mail needs to connect to ASSP on port 25 (ASSP's listen port), and ASSP can proxy to your mail server on port 125 (or any port you choose) -- ASSP's SMTP Destination. You need to change your mail server to match.
  • 42. Sender Notification
    • With most client-based filters (POPFile, SpamBayes, SpamAssassin) senders receive NO NOTIFICATION if their mail isn't delivered. With most of these solutions, the user bears full responsibility to VERIFY that no good mail is blocked.
    • ASSP’s solution to this is that when spam is blocked the SENDER RECEIVES NOTIFICATION, and it does this without generating non-delivery reports that bounce and bounce again because spammers forge their from address.
  • 43. Catch-22
    • Issue: Let’s say a client receives a non-delivery report, how can he (not in whitelist) send a message to the organization if he is still not in whitelist? I mean, if the recipient or assp admin does not receive the notification, they will not know that there is a false positive and will not add the unknown client to whitelist...
    • Solution: Set up an email address and put it in the Spam-Lover Address configuration option. Then modify the spam error message to direct people to &quot;500 Mail appears to be unsolicited (spam) -- please forward this email to not-spam@mydomain.com if you feel this is in error.&quot; Any false positives that bounce back to clients will hopefully be reported to the Mail Admin via the spam lover address (they just forward it), assuming they read the rejected email.
  • 44. Email Interface
    • Any user can help to improve ASSP’s spam filtering accuracy. Users can use it to add addresses to the whitelist, report spam, or false-positives. To use it, you must have it enabeled in the configuration, and have names set for the addresses. The interface only accepts mail addressed to addresses at any of your localdomains, and only from &quot;Accept All Mail&quot; hosts, or authenticated SMTP connections.
    • assp-white -- for whitelist additions
    • assp-spam -- to report spam that got through
    • assp-notspam -- to report mis-categorized spam
    • Whitelisting: Assuming that your local-domain is yourdomain.com, to add addresses to the whitelist, you’d create a message to [email_address] . You can either put the addresses in the body of the message, or as recipients of the message. For example, if you wanted to add all the addresses in your address book to the whitelist, create a message to [email_address] and then add your entire address book to the BCC part of the message and click send. Note that no mail will be delivered to any address except [email_address] (and that won't actually be passed to your mail transport). Within a short time you'll receive a response from ASSP showing the results of your mail.
    • False Negatives: To report a spam that got through, simply forward the mail to [email_address] . It's best to forward it as an attachment, but you can just forward it normally if you must. In a short time you will receive a confirmation.
    • False Positives: The process is the same to report a miscategorized spam, but send it to [email_address] .
  • 45. Spam Report
  • 46. Benchmarks
    • Spam Bucket
    • Ex-employee that left the company 5 years ago
    • Receives 50-80 spam mails per day
  • 47. Filter effectiveness
    • SpamAssassin 60-65% effective in 2004
    • Deteriorated to 11% by 2006
    • (267 of 2238 True Positives)
    • ASSP in first 3 weeks of operation 99.7%
    • (1336 of 1340 True Positives)
  • 48. ASSP vs SpamAssassin
    • SpamAssassin
      • is difficult to install
      • great investment in hand-made regular expressions and header analysis to identify spam
      • Hand-crafted expressions are brittle as spammers adjust their strategies
      • Requires frequent updates to accurately identify spam
    • ASSP
      • is low maintenance
      • is easy to install
      • is a complete spam blocking solution, not just a filter that must be integrated into your MTA
      • works with nearly every MTA on any OS
      • Poorly documented
  • 49. Before ASSP
  • 50. Turning ASSP on
  • 51. With ASSP
  • 52. stat.pl Statistics [root@smtp]# perl stat.pl /tmp/m.log As of Mon Jan 22 21:48:46 2007 the mail logfile shows: 0 proxy / smtp connections 253 were dropped for attempted relays (0.0% of total). 31523 messages, 16758 were spam (53.2%) in 65 days for 485.0 messages per day or 257.8 spams per day 1518 additions to / verifications of the whitelist (23.4 per day) 14643 were judged spam by the bayesian filter (87.4% of spam) 2115 were to spam addresses (12.6% of spam) 0 were rejected for executable attachments (0% of spam) 10121 were sent from local clients (68.5% of nonspam) 842 were from whitelisted addresses (5.7% of nonspam) 0 messages were passed to SPAMLOVERs 3802 were ok after a bayesian check (25.8% of nonspam) 1498 addresses are on the whitelist 0 hits on the blacklist 0 resulted in spam (0.0% of Bayesian spam, 0.0% of blacklist hits) 0 resulted in non-spam (0.000% of blacklist hits)
  • 53. ASSP Statistics
  • 54. Issues
    • Vacation
    • Auto Replies
    • TLS and secure SMTP
    • ASSP is site based, not per-user
  • 55. Lessons Learned
    • Whitelist + spambucket + Bayesian is a great spam filtering strategy
    • The default is SPF failures will filter even if whitelisted
    • Be very careful what you put in the relay hosts list
    • ASSP is not multi-process or multi-threaded
  • 56. Utilities
    • rebuildspamdb.pl
    • repair.pl
    • move2num.pl
    • stat.pl
  • 57. Demo
    • Web configuration
    • Mail analyzer
  • 58. Resources on the Internet
    • http://www.spamland.com
    • http://antispam.yahoo.com
    • http://www.openspf.org
  • 59. Questions