ScavengerEXAAn open source programTo fight spam at the sourceThomas ManginExa NetworksUKNOF 122009/02/13http://wiki.exa.org.uk/doku.phpdo=export_s5&id=scavenger:uknof12
The spam battle - YesterdayMouse:Using existing mail servers, open relayFew high volume sourceCat:scanning for open relayUsing RBL-> Trying to block the spam at the source
The spam battle - TodayMouse:BOTNETs, Spammers creating their own spaminfrastructureLots of “low” volume sourceCat:Bots are simple, not RFC compliant (Greylisting)Spam traps -> RBLswhen all fails ask SpamAssasin for some CPU time-> The defense is now at the receiving end
The spam battle - TomorrowMouse:Larger, more clever bots, able to bypass greylistingBetter spam distribution to become more stealthyCat:Trying to block bots faster, so they send less-> It looks bleak, doesnt it ?
Using my crystal ballWhat will not change:The use of botsThe use of out of date email address databasesCompromised machine sending spam but no mailbefore or after.What should change:Postmaster should get help some help from thenetwork, spammers do.The fight should be brought back to the edge
Why am I here ?1- present ScavengerEXA2 - reduce the spam in //my// mailbox3 - ?4 - profit .. not !Get help from the community to produce a turn keysolutionConvince some to help us with an high profile deployment
What is scavengerEXAOn the net:http://www.thefreedictionary.com/scavengerAlteration of Middle English scauager, schavager, officialcharged with street maintenanceIn my book:An carnivore eating Junk created by Exahttp://en.wikipedia.org/wiki/Carnivore_(FBI)http://en.wikipedia.org/wiki/Junk_mailhttp://en.wikipedia.org/wiki/Exa-
ScavengerEXA DesignA collection of several application workingtogether through the networkmail conversation capture programdispatch serverpolicy server (postfix policy server alike)action servers (email, block spam, etc. the partno ISP wants the same)a dummy MTA (return 450 on all messages)
Capturelibpcap based applicationkeep a track of the smtp conversation:client command, parameterEHLO [127.0.0.1]RCPT TO: <firstname.lastname@example.org>mail server answers250 Please to meet you550 user does not exists hereIgnore the body of the mailTransmit a UDP packet for each command to the dispatchingserver
UDP Message Contentkey=value structure, with the following keys (in no particularorder) for each unique si:sp -> di:dpor=pacp (how the packet was created)in=unique random id (identifying the SMTP conversation)si=source IP, the potential botdi=destination IP, mail server contactedhe=EHLO/HELO stringst=state (HELO,MAIL,RCPT,DATA,END-OF-DATA)re=last recpient email addressrc=number of recpient in the mailse=sender email addressco=smtp response code
ExampleMail conversation220 mail server listingHELO [127.0.0.1]250 mx.domain.comMAIL FROM: email@example.com 2.1.0 OkRCPT TO: firstname.lastname@example.org 5.7.1 <email@example.com>: Recipient addressrejected: no such user<Disconnection>We do not track smtp auth information atm
UDP Message sent 1/3HELO [127.0.0.1]250 mx.domain.comor=pcapco=250di=18.104.22.168rc=0st=EHLOre=si=22.214.171.124in=01.3c6b.4e.c0c3se=he=[127.0.0.1]
UDP Message sent 2/3MAIL FROM: firstname.lastname@example.org 2.1.0 Okor=pcapco=250di=126.96.36.199rc=0st=MAILreemail@example.com=[127.0.0.1]
UDP Message sent 3/3RCPT TO: firstname.lastname@example.org 5.7.1 <email@example.com>: Recipient addressrejected: no such useror=pcapco=550di=188.8.131.52rc=1st=RCPTrefirstname.lastname@example.orgemail@example.com=[127.0.0.1]
Dispatch Logic1 - Receive the UDP message from a capture sourceFigure out a policy server (hash on sender IP)2 - Generate a TCP message for that policy server withthe same information as the UDP contained3 - Wait for the answer from the policy server4 - If the policy server reports it is spam5 - Generate a/several TCP message(s) to actionserver(s)Prevent new message to the Policy Server from thespammer
Policy DaemonBased on the code of Exas internal Postfix PolicyDelegation Daemon (the daemon can still be used withPostfix)Use the UDP message format instead of PostfixsReturns:HAMHOLD <IP> (<duration>) <reason>FILTER <IP> [<MTA IP:PORT>] (<duration>) <reason>an MTA of [0.0.0.0:00000] mean that the MTA used tofilter the message is left to the software performing theblocking to decide.
Policy DaemonA clear API for its spam classfication pluginswith constraint, run ifthe message match what you are monitoringthe backend DB is MySQL ...Explaining its design would take another 20 slideswriting plugins is **//simple//**Each plugin has its own database connection.support MySQL, PostgreSQL or Sqlite3
Action MailYou may want to use this action whatever elseyou do.Takes a HOLD or FILTER messageEmails you the suspected spammer information
Action NetfilterRead a Linux machine netfilter rules.Force Transproxying of mail to a specified MTAon receipt of a FILTER ruleRemove the FILTERING when the filtering periodis overKeep state across reboots
It is F.R.E.ELicensed under the Affero GPL 3.0Each ScavengerEXA program being a separateentity, there is no problem implementing a newserver under a proprietary licence.AFAIK, IMNAL
It is available nowPublic SVN Treehttp://svn.exa.org.uk/scavenger/trunkFisheyehttp://fisheye.exa.org.uk/browse/scavenger/provides nice delta of changesRSS feed of commitsMailing listhttp://mailman.exa.org.uk/mailman/listinfo
Build on solid foundationsPython, tested with version 2.5.2Twisted Matrix, event-driven networking enginePython packet capture library (pcap)Python dump packet module (dpkt)Optional MySQL or PostgreSQLFor more information see:http://wiki.exa.org.uk/scavenger/faq
Thank you toRichard Clayton SpamHINTShttp://www.spamhints.org/
Questions ?Want to know more:website: http://scavenger.exa.org.uk/Contact us:email: scavenger (at) exa (dot) org (dot) ukThank you for listening.