Uploaded on

An attempt to create a spam detection framework working near the source of spam.

An attempt to create a spam detection framework working near the source of spam.

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
    Be the first to like this
No Downloads


Total Views
On Slideshare
From Embeds
Number of Embeds



Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

    No notes for slide


  • 1. ScavengerEXAAn open source programTo fight spam at the sourceThomas ManginExa NetworksUKNOF 122009/02/13
  • 2. The spam battle - YesterdayMouse:Using existing mail servers, open relayFew high volume sourceCat:scanning for open relayUsing RBL-> Trying to block the spam at the source
  • 3. The spam battle - TodayMouse:BOTNETs, Spammers creating their own spaminfrastructureLots of “low” volume sourceCat:Bots are simple, not RFC compliant (Greylisting)Spam traps -> RBLswhen all fails ask SpamAssasin for some CPU time-> The defense is now at the receiving end
  • 4. The spam battle - TomorrowMouse:Larger, more clever bots, able to bypass greylistingBetter spam distribution to become more stealthyCat:Trying to block bots faster, so they send less-> It looks bleak, doesnt it ?
  • 5. Using my crystal ballWhat will not change:The use of botsThe use of out of date email address databasesCompromised machine sending spam but no mailbefore or after.What should change:Postmaster should get help some help from thenetwork, spammers do.The fight should be brought back to the edge
  • 6. Why am I here ?1- present ScavengerEXA2 - reduce the spam in //my// mailbox3 - ?4 - profit .. not !Get help from the community to produce a turn keysolutionConvince some to help us with an high profile deployment
  • 7. What is scavengerEXAOn the net: of Middle English scauager, schavager, officialcharged with street maintenanceIn my book:An carnivore eating Junk created by Exa
  • 8. ScavengerEXA DesignA collection of several application workingtogether through the networkmail conversation capture programdispatch serverpolicy server (postfix policy server alike)action servers (email, block spam, etc. the partno ISP wants the same)a dummy MTA (return 450 on all messages)
  • 9. Capturelibpcap based applicationkeep a track of the smtp conversation:client command, parameterEHLO []RCPT TO: <>mail server answers250 Please to meet you550 user does not exists hereIgnore the body of the mailTransmit a UDP packet for each command to the dispatchingserver
  • 10. UDP Message Contentkey=value structure, with the following keys (in no particularorder) for each unique si:sp -> di:dpor=pacp (how the packet was created)in=unique random id (identifying the SMTP conversation)si=source IP, the potential botdi=destination IP, mail server contactedhe=EHLO/HELO stringst=state (HELO,MAIL,RCPT,DATA,END-OF-DATA)re=last recpient email addressrc=number of recpient in the mailse=sender email addressco=smtp response code
  • 11. ExampleMail conversation220 mail server listingHELO []250 mx.domain.comMAIL FROM: test@domain.com250 2.1.0 OkRCPT TO: user@spammed.com550 5.7.1 <>: Recipient addressrejected: no such user<Disconnection>We do not track smtp auth information atm
  • 12. UDP Message sent 1/3HELO []250 mx.domain.comor=pcapco=250di=[]
  • 13. UDP Message sent 2/3MAIL FROM: test@domain.com250 2.1.0 Okor=pcapco=250di=[]
  • 14. UDP Message sent 3/3RCPT TO: user@spammed.com550 5.7.1 <>: Recipient addressrejected: no such useror=pcapco=550di=[]
  • 15. Dispatch Logic1 - Receive the UDP message from a capture sourceFigure out a policy server (hash on sender IP)2 - Generate a TCP message for that policy server withthe same information as the UDP contained3 - Wait for the answer from the policy server4 - If the policy server reports it is spam5 - Generate a/several TCP message(s) to actionserver(s)Prevent new message to the Policy Server from thespammer
  • 16. Message Flow
  • 17. Policy DaemonBased on the code of Exas internal Postfix PolicyDelegation Daemon (the daemon can still be used withPostfix)Use the UDP message format instead of PostfixsReturns:HAMHOLD <IP> (<duration>) <reason>FILTER <IP> [<MTA IP:PORT>] (<duration>) <reason>an MTA of [] mean that the MTA used tofilter the message is left to the software performing theblocking to decide.
  • 18. Policy DaemonA clear API for its spam classfication pluginswith constraint, run ifthe message match what you are monitoringthe backend DB is MySQL ...Explaining its design would take another 20 slideswriting plugins is **//simple//**Each plugin has its own database MySQL, PostgreSQL or Sqlite3
  • 19. Action MailYou may want to use this action whatever elseyou do.Takes a HOLD or FILTER messageEmails you the suspected spammer information
  • 20. Action NetfilterRead a Linux machine netfilter rules.Force Transproxying of mail to a specified MTAon receipt of a FILTER ruleRemove the FILTERING when the filtering periodis overKeep state across reboots
  • 21. Deployment Example
  • 22. Deployment Example
  • 23. It is F.R.E.ELicensed under the Affero GPL 3.0Each ScavengerEXA program being a separateentity, there is no problem implementing a newserver under a proprietary licence.AFAIK, IMNAL
  • 24. It is available nowPublic SVN Tree nice delta of changesRSS feed of commitsMailing list
  • 25. Build on solid foundationsPython, tested with version 2.5.2Twisted Matrix, event-driven networking enginePython packet capture library (pcap)Python dump packet module (dpkt)Optional MySQL or PostgreSQLFor more information see:
  • 26. Thank you toRichard Clayton SpamHINTS
  • 27. Questions ?Want to know more:website: us:email: scavenger (at) exa (dot) org (dot) ukThank you for listening.