Creating an Open Source Email Filtering Gateway - a How-To by Larkin Cunningham


Published on

Creating an Open Source Email Filtering Gateway - a How-To by Larkin Cunningham that appeared in Linux Magazine

  • Be the first to comment

  • Be the first to like this

Creating an Open Source Email Filtering Gateway - a How-To by Larkin Cunningham

  1. 1. COVER STORY Amavisd-new Filtering spam and viruses at the mail server with Amavisd-new SCAN MANAGER Deutsche Post WorldNet Sometimes the best time to stop bad mail is before it arrives. Amavisd- new is a Open Source interface for integrating spam and virus filtering a daemonized set of master and child with your mail server. BY LARKIN CUNNINGHAM processes. Amavisd-new acts as an SMTP server, receiving email messages from your SMTP server (e.g., Postfix, exim, or qmail), processing the messages, and T he majority of viruses are propa- duces the risk of spam and viruses sending them or reinjecting them back gated via email. While it is still reaching your desktop. An additional ad- into your SMTP server. possible to receive viruses from vantage of providing this extra layer of Amavisd-new supports anti-spam floppy disks and CD-ROMs, or from In- protection is that, by using a virus tools such as SpamAssasin [2], as well ternet worms and root kits that directly checker on your desktop that is different as a wide range of commercial and open attack vulnerabilities in your operating from the content checker on your SMTP source virus checkers. The popular Clam system, you are much more likely to re- server, you reduce the risk of a virus Antivirus [3] is supported in three ways; ceive viruses via email – often by inad- passing undetected. the clamd daemon (the best perfor- vertently opening seemingly innocent at- Amavisd-new [1] is an Open Source mance), the Mail::ClamAV Perl package tachments. tool that serves as an interface from a (not so good performance) and the clam- Of course it is important to have an mail server to virus scanners and other scan command line option (as a backup up-to-date virus checker on your desk- forms of content checkers. Although when clamd is unavailable, for exam- top, but defense in depth is the best ap- some virus tools provide their own ple). Many other popular virus checkers proach, and this is why you should be mechanisms for filtering mail at the are also supported, including F-Prot, protected from spam and viruses at your server, Amavisd-new offers performance Sophos, Grisoft’s AVG, KasperskyLab receiving SMTP server. Virus protection advantages in some environments, and AVP, AntiVir, F-Secure, McAfee and at the mail server prevents undesirable it also provides a single, vendor-neutral Panda. But be aware of the licensing for email from reaching your desktop, and it configuration point for managing both the product you choose. Obviously also keeps viruses out of your POP3 or virus and spam filtering. ClamAV will not incur any license fee, IMAP account. Filtering from the mail but some of the other options may have High Performance server reduces the load on your desktop a license fee for an SMTP server that is PC or laptop, reduces the amount of Amavisd-new is a flexible, high perfor- significantly greater than the desktop li- bandwidth you will use, and greatly re- mance Perl-based application that runs as cense fee. 35 ISSUE 62 JANUARY 2006 W W W. L I N U X - M A G A Z I N E . C O M
  2. 2. COVER STORY Amavisd-new You can configure Amavisd-new to bases are also sup- block attachments with commonly dan- ported, including gerous extensions, such as .exe, .bat, Oracle and DB2. and .vbs (particularly dangerous to Win- Be aware that dows clients who may be using the mail many of the tables server). You can also specify a wide have serial / se- range of decoders/decompressors to ex- quence primary amine archives such as .cpio, .rpm, .deb, keys, so for Ora- . zoo, .tar, .gz, and .bz2. cle, you’ll need to Amavisd-new can theoretically sup- add several se- port any SMTP server, but it works bet- quences and trig- ter with the usual suspects, including gers in addition to Sendmail, exim, and qmail. Amavisd- the tables. new works best with Postfix [4], which You must spec- can reinjectï¿œmail back into itself after ify two DSNs content filtering. (Data Source Figure 1: Amavisd-new is just one big Perl script. Names). One DSM Installation Requirements you have set up the background applica- is for lookups and the other for storage To install Amavisd-new, you need a tions, the installation is relatively straight- (logging). You can improve performance working Perl installation. Perl 5.8.2 or forward. Amavisd-new does not require by using a database that is very fast for greater is recommended. Though earlier the compilation of any code, since it uses read-only access to do lookups and a da- versions of Perl will work, later versions the Perl interpretor. tabase that is fast for writes for storage. work better. You need to install a num- Amavisd-new also lets you specify multi- Configuring Amavisd-new ber of Perl packages (Table 1). Be sure to ple DSNs to allow for failover. with SQL update any packages you already have to Policies the latest versions. There are several ways to configure Obviously, for effective spam filtering, Amavisd-new. You can define your Ama- If you are familiar with firewall configura- the latest Mail::SpamAssassin package visd-new configuration using the ama- tion, you will be familiar with the term should be installed. Spammers attempt visd.conf file, lists files, hash lookup policy. A firewall policy is a set of instruc- to stay one step ahead of SpamAssassin, files, regular expressions, and LDAP or tions to a firewall specifying what to do so you should always update the SQL lookups. with certain types of network traffic. For SpamAssassin package as soon as a new The SQL solution gives the greatest example, a firewall policy might specify version becomes available. opportunity to build a front-end configu- whether to block traffic to a certain net- You should also install a number of ad- ration tool using a scripting language work port or whether to redirect traffic ditional programs to allow for the widest such as PHP. You can use any database from one port to a different port. An range of decoding and decompressing of that supports standard SQL and is sup- Amavisd-new policy is similar. With each email attachments. These additional pro- ported by Perl DBD/DBI libraries. SQL policy, you can direct Amavisd-new to grams include gzip, bzip2, arc, lha, rar, DDL (Data Definition Language) is pro- bypass spam checking or virus filtering, zoo, pax, cpio, freeze, ripole, cabextract, vided in the README.sql file to create be tougher or more lenient with spam, and many others. See the Amavisd-new tables, indexes, and some sample data. tag the spam email’s subject line with a website [1] for more on installation. After Notes are included for MySQL, Post- marker (like {Spam?}), and forward all greSQL, and SQLite. Many other data- spam to a designated spam account. Table 1: Required Perl Table 2: Some of the options in the policy table Packages Archive::Tar virus_lover Do not reject the email even if virus infected Archive::Zip spam_lover Do not reject the email even if identified as spam Compress::Zlib Note: The _lover options do not tell Amavisd-new not to scan, just to ignore the results of the scan. Convert::TNEF bypass_virus_checks Do not scan the email for viruses Convert::UUlib bypass_spam_checks Do not scan the email for spam and do not add the X-Spam headers MIME::Base64 spam_modifies_subj Add some text (a tag) to the beginning of the subject if it is spam MIME::Parser virus_quarantine_to An email account to send all virus infected emails to Mail::Internet spam_quarantine_to An email account to send all spam to Net::Server spam_tag_level The SpamAssassin score above which spam emails get X-Spam headers Net::SMTP spam_tag2_level The SpamAssassin score above which spam gets the subject tagged Digest::MD5 spam_kill_level The SpamAssassin score above which ‘evasive’ action is taken, determined by $final_spam_destiny in the configuration file (default is IO::Stringy to discard the email, i.e. Send it to a blackhole) Time::HiRes spam_subject_tag The text to use for the spam tag (see spam_modifies_subj) Unix::Syslog spam_subject_tag2 The text to use for the high scoring spam tag BerkeleyDB 36 ISSUE 62 JANUARY 2006 W W W. L I N U X - M A G A Z I N E . C O M
  3. 3. COVER STORY Amavisd-new Black or white listed emails are added to Logging and the Law the mailaddr table with a unique identi- A hot topic for ISPs at the moment is a fier. The rows in the wblist table specify potential requirement to store email the email address and target user to data for up to three years. This is in re- which the rule applies. sponse to recent terrorist atrocities and is designed to give law enforcement Logging and Archiving agencies information to track the activi- One of the newer features of Amavisd- ties of terrorists, in addition to telephone new is the ability to log all email activity. and mobile phone call and SMS logs. The logging information is comprehen- While storing email content would inevi- sive and includes information such as tably lead to a massive storage burden for ISPs, the archiving of the logging in- senders, recipients, times, dates, sub- formation generated by Amavisd-new jects, and spam or virus scores. Loging Figure 2: Amavisd-new provides detailed could address many of the requirements lets you keep accurate statistics on the logging information that allows for informa- of any future European or local govern- level of clean versus spam versus virus tive charts. ment legislation. emails received. The fact that you can record these statistics at the user level taking a look at the Postfix and Amavisd- Each email domain or user can have gives the system administrator an oppor- new combination. Particularly for ISPs, its own policy. You can specify a policy tunity to see where content checking re- there is an imperative to providing cus- to cover all email accounts within a do- sources are most needed. For an ISP (In- tomers with spam and virus filtering, main and also decide to have additional ternet Service Provider), this gives the whether it is free or as an add-on to their policies for individual email accounts opportunity to bill for bandwidth, pro- service. within the same domain. For example, cessing time, number of messages pro- Growing levels of spam and viruses you might have a default policy to cover cessed, or the number of spam or virus and greater numbers of customers are all users in a domain (like a catch-all emails processed. putting a strain on the infrastructure of policy), but perhaps Tom’s email ac- many ISPs. An open source solution Conclusion count is being flooded with spam. An in- using Postfix and Amavisd-new could be dividual policy for Tom’s account could For those who are considering building a the answer. Both Postfix and Amavisd- have a lower tolerance. Table 2 details robust and scalable architecture for bat- new can be configured to use MySQL, just some of the policy options. tling spam and viruses, I recommend PostgreSQL, or Oracle for retrieving data. A user table allows you to assign poli- It is possible to have a central database Administration and cies to specific domains and individual accessed by multiple Postfix and Ama- Statistics Made Easy email accounts. The user table’s email visd-new servers. And it is possible to field determines whether multiple email employ multiple databases, allowing for Using a database instead of configura- accounts or just a single email account greater fault tolerance. By using a num- tion files allows for easy administration. will be affected by the policy. For exam- ber of failover techniques, including The database option also allows for dy- ple, the email refers to all configuration of DNS records, it is possi- namic updating of configuration data email accounts in the do- ble to build a scalable, fully fault toler- without having to reload the Amavisd- main. But, refers to a ant, and load balanced infrastructure. ■ new daemons. The same is true if you opt to configure your Postfix users and single email address. If both @domain. INFO domains in a SQL database. By query- com and exist in the ing the storage database using SQL, it is users table, precedence is given to the [1] Amavisd-new: easy to garner useful information on specific email address first. spam and virus statistics. You could [2] SpamAssassin: achieve this using phpMyAdmin, for ex- Black and White Listing ample, if MySQL is the chosen database. When important emails you want to re- [3] Clam Antivirus: An ISP with a large number of custom- ceive continually get blocked because ers could develop an interface that [4] Postfix SMTP Server: they are identified as spam, you can take would allow customers to update their action to ensure that the sender of the own policies to be more or less aggres- emails is exempt from spam checking by sive against spam or viruses. It would adding them to the white list. Con- Larkin Cunningham is an IT Consul- also allow users to view statistics on the versely, when you continually receive THE AUTHOR tant specializing in the areas of levels of clean, spam, and virus infected Linux Administration, Java, Applica- spam from a source you know is a spam- emails. These types of statistics can only tion Frameworks, Oracle and just help in the education of email users. Fig- mer, you can add the email address to about anything in the open source ure 2 shows an example of the type of the black list. community that makes the lives of statistics you can easily present to cus- Amavisd-new’s black and white lists IT Managers easier and Finance Di- tomers. The example was developed work by cross referencing the mailaddr rectors happier. He can be contacted using ColdFusion MX 7, but you could (email address) table with the user table at also use PHP with jpGraph. and the wblist (white / black list) table. 37 ISSUE 62 JANUARY 2006 W W W. L I N U X - M A G A Z I N E . C O M