You had me at HELO

A case study in spam fighting
Randal L.  Schwartz

Stonehenge Consulting Services,  Inc. 
mer(DELETETH|...
What is spam? 

Unwanted mail
 as I defined “unwanted”

Usually sales pitches

.  Occasionally virus/ worm payloads
First amendment

0 I will defend your right to say what you
want to say. ..

0  but not at my expense

0 “Freedom of the p...
My role

Own a small business

Therefore,  the online presence manager
Actually,  more of a janitor

Single machine,  leas...
B. S. (before spam)

Could put an email address on a web page
Or in a Usenet posting
Or in a mailing list that got archive...
A. S.S.  (after spamming
started)

II If you use an email address on a web page, 

or in Usenet posting. ..

I5  you can e...
But it gets worse

I” Apparently,  80 million addresses weren’t
enough

IE Dictionary attacks

III Address-book scraping w...
And worse

III “Joe jobbing” and friend-spam to get past
human filters,  using false “From” lines

I‘ But this also trigger...
The first step

Mail for $user@stonehenge. com came in
to my email via a procmail script

In 2002, I added a SpamAssassin c...
Then came the MIRVS

III Multiple Independently targetable Reentry
Vehicles

'3 Or in my case,  spam addressed to multiple...
Why MIRVS hurt

III They came in clusters:  my MTA delivered
them all in a row quickly

I3 They launched separate Perl inv...
The next step

In mid 2003, I moved SpamAssassin into the
MTA (Postfix),  before the delivery queue

Amavisd keeps code in ...
A bit of relief

For a while,  life was good

From about September 2003 to February
2004

Spam slowly increased,  but mang...
Worms,  ick! 

Windows infestations
Very rapidly spread

Many variants (antivirus people fighting to
keep up)

.  Included ...
Worm mail

III Fat payload
Ii Rapid fire (many connections at once)
I3 Ultimate purpose: 
II To spread

II To create millio...
My response

Panic! 
Filter the worm payloads with procmail

This kept my personal mail sorter from

being invoked on worm...
New problem: 
bandwidth

III Load average was marginal,  but now my
port 25 bandwidth far exceeded my web
traffic

III Push...
Fight back with filtering

iii OpenBSD gives me pf:  a very powerful
packet filter and traffic shaper

*5 IP lists can be dy...
Bad Boys,  Bad Boys

i Block the people being bad to me

it Anyone sending mail to a known
“spamtrap” mail address in ston...
The details

Perl process watching logs from Postfix, 
Amavisd,  and procmail

All first-hop IP addresses found there
added ...
The results

Email bandwidth reduced by half
Now paralleled the web traffic

Barely within purchased limits

.  But at leas...
Why stop there? 

OpenBSD packet filter includes passive
operating-system fingerprinting

Based on p0f code

- Windows is ev...
Why slow and not
stop? 

Because some of my friends actually run
Windows and want to send me email

Because some corporate...
Who was I blocking? 

Packets being blocked are logged to pflog0
Watch fingerprints with tcpdump

Bad boys still attempting...
More bad boys

I realized that Postfix could block SMTP
connections from unknown hosts

Unknown hosts have no valid reverse...
The downside

Reverse DNS should be set up for all
corporate mail gateways

But some Very Large Companies still aren’t
pay...
High-MX spamtrap

Legit mail should be delivered to low MX

Some spam software delivers to higher MX
values first (or only)...
My implementation

Added an alternate IP to my box
Gave it a name,  and a high MX value

Taught Postfix to listen separatel...
Caution about this
spamtrap

it Must make sure that high MX wasn’t a
legitimate sender rolling over because low
MX was hos...
Results of High-MX
spamtrap

Implemented only recently
Appears to be blocking 90 messages/ hr

No false positives seen! 

...
Sharing the knowledge

it Publishing my bad-boys list experimentally
as an RBL at rb| .stonehenge. com,  updated
every min...
To-do,  to-don’t

it Tried and abandoned HELO-based domain

verification blocks;  too many machines are
misconfigured

‘I Lo...
The cost of
spamfi ghtin g
Many hours of labor to keep things updated

Bandwidth overages

Excessive CPU load averages

.  ...
My dream

0 Make junk email illegal (like junk fax/ cell)

0 Culpability for Operating System Producers
(read: “microsoft”...
More resources

0 Magazine articles I've written (Google for
“site: stonehenge. com spam”)

0 Documentation for Postfix, Am...
Upcoming SlideShare
Loading in …5
×

You had me at HELO: A case study in spam fighting by Randal L.Schwartz

1,969 views
1,815 views

Published on

http://www.stonehenge.com/merlyn/Pictures/Trips/2004/04-09-DragonCon/HELO.pdf

Published in: Economy & Finance, Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
1,969
On SlideShare
0
From Embeds
0
Number of Embeds
44
Actions
Shares
0
Downloads
0
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

You had me at HELO: A case study in spam fighting by Randal L.Schwartz

  1. 1. You had me at HELO A case study in spam fighting Randal L. Schwartz Stonehenge Consulting Services, Inc. mer(DELETETH| S)lyn AT Stonehenge DOT com
  2. 2. What is spam? Unwanted mail as I defined “unwanted” Usually sales pitches . Occasionally virus/ worm payloads
  3. 3. First amendment 0 I will defend your right to say what you want to say. .. 0 but not at my expense 0 “Freedom of the press” doesn't mean everyone gets a free press 0 and especially not you using my press to try to sell me something
  4. 4. My role Own a small business Therefore, the online presence manager Actually, more of a janitor Single machine, leased at a co-lo Biggest expense: web bandwidth in theory
  5. 5. B. S. (before spam) Could put an email address on a web page Or in a Usenet posting Or in a mailing list that got archived without worrying about it being spammed to death
  6. 6. A. S.S. (after spamming started) II If you use an email address on a web page, or in Usenet posting. .. I5 you can expect hundreds or thousands of messages per day III Fewer (perhaps none) if you can entitize or use “illegal characters” *9 But once it’s in the wild, it’s permanent
  7. 7. But it gets worse I” Apparently, 80 million addresses weren’t enough IE Dictionary attacks III Address-book scraping worms
  8. 8. And worse III “Joe jobbing” and friend-spam to get past human filters, using false “From” lines I‘ But this also triggers false replies from “helpful” (actually harmful! ) anti-spam and anti-virus filters Ii Important note: do not bounce spam. Ever. Ever! Just drop it. Please. Please!
  9. 9. The first step Mail for $user@stonehenge. com came in to my email via a procmail script In 2002, I added a SpamAssassin check, looking at content and RBLs And instantly, my unwanted mail went to about a third of what it was! Yeay!
  10. 10. Then came the MIRVS III Multiple Independently targetable Reentry Vehicles '3 Or in my case, spam addressed to multiple users within the stonehenge. com domain. If One SMTP transaction, but many separate procmail deliveries
  11. 11. Why MIRVS hurt III They came in clusters: my MTA delivered them all in a row quickly I3 They launched separate Perl invocations for SpamAssassin III Result: at first, load average spikes, then continuously bad load
  12. 12. The next step In mid 2003, I moved SpamAssassin into the MTA (Postfix), before the delivery queue Amavisd keeps code in preforked memory Also (most important) handles a MIRV before it bursts And, can reject spam during the SMTP handshake, so no need to bounce spam
  13. 13. A bit of relief For a while, life was good From about September 2003 to February 2004 Spam slowly increased, but mangeably And then, “invasion of the worms”
  14. 14. Worms, ick! Windows infestations Very rapidly spread Many variants (antivirus people fighting to keep up) . Included their own MTA, with its own rules about connections and delivery rates
  15. 15. Worm mail III Fat payload Ii Rapid fire (many connections at once) I3 Ultimate purpose: II To spread II To create millions of spam relays controlled by private enterprise
  16. 16. My response Panic! Filter the worm payloads with procmail This kept my personal mail sorter from being invoked on worm payloads, saving me some CPU Tried to teach SpamAssassin about worm payloads, but failed
  17. 17. New problem: bandwidth III Load average was marginal, but now my port 25 bandwidth far exceeded my web traffic III Pushed me into overage charges III Thank you sprocketdata. com for not actually charging me anything!
  18. 18. Fight back with filtering iii OpenBSD gives me pf: a very powerful packet filter and traffic shaper *5 IP lists can be dynamically updated and efficiently scanned *9 My goal: if someone is bad to me, then they only get one chance to be bad *9 So I added a port 25 filter
  19. 19. Bad Boys, Bad Boys i Block the people being bad to me it Anyone sending mail to a known “spamtrap” mail address in stonehenge. com (most| y“thanks” to the evil henge. com) 4! Anyone sending spam ‘E Anyone sending worm payloads
  20. 20. The details Perl process watching logs from Postfix, Amavisd, and procmail All first-hop IP addresses found there added to the port 25 block via pfctl Automatically removed after two hours Removal permits legit mail to flow again Most worms don’t retry once blocked
  21. 21. The results Email bandwidth reduced by half Now paralleled the web traffic Barely within purchased limits . But at least this was livable At any given moment, 750 to I500 IP addresses in the bad boy list
  22. 22. Why stop there? OpenBSD packet filter includes passive operating-system fingerprinting Based on p0f code - Windows is evil So slow down all the windows email! Collective speed of all windows machines reduced to a dialup-modem speed!
  23. 23. Why slow and not stop? Because some of my friends actually run Windows and want to send me email Because some corporate mail gateways run Windows Net effect: most worms were slowed, but legit mail continued to trickle through All managed by pf: no work on my part
  24. 24. Who was I blocking? Packets being blocked are logged to pflog0 Watch fingerprints with tcpdump Bad boys still attempting to reconnect about 30 to 90 times a minute . Almost all connections are from Windows machines, probably wormed
  25. 25. More bad boys I realized that Postfix could block SMTP connections from unknown hosts Unknown hosts have no valid reverse DNS I added that block in mid-2004, and email was reduced a whopping additional 70%! Once again, SMTP traffic was dwarfed by web traffic! Load averages were normal!
  26. 26. The downside Reverse DNS should be set up for all corporate mail gateways But some Very Large Companies still aren’t paying attention So, I had to add a whitelist for the broken sites, and tools to watch the blocked list This extra workload is worth the result
  27. 27. High-MX spamtrap Legit mail should be delivered to low MX Some spam software delivers to higher MX values first (or only), violating RFCs Secondary machines rarely config’ed right This usually gets around spamblocks But this can also be a very solid spamsign
  28. 28. My implementation Added an alternate IP to my box Gave it a name, and a high MX value Taught Postfix to listen separately to the primary address and spamtrap address Postfix sends back a “450 Violation of RFC282| Section 5 paragraph 8” for all connections to the spamtrap port 25
  29. 29. Caution about this spamtrap it Must make sure that high MX wasn’t a legitimate sender rolling over because low MX was hosed (same host in my case) *1 Return 450 code, so that a legit sender will retry, hopefully getting the non-spamtrap the next time *3 Most spam senders will just go away
  30. 30. Results of High-MX spamtrap Implemented only recently Appears to be blocking 90 messages/ hr No false positives seen! . Also adding these to my bad-boy list, which foils spamsender from trying lower MX So, it’s probably even better than it looks
  31. 31. Sharing the knowledge it Publishing my bad-boys list experimentally as an RBL at rb| .stonehenge. com, updated every minute '3 Please ask before using permanently 1* Publishing list at http: // www. stonehenge. com/ pic/ rb| .stonehenge. com. txt, updated every minute
  32. 32. To-do, to-don’t it Tried and abandoned HELO-based domain verification blocks; too many machines are misconfigured ‘I Look at longer history of lPs that repeatedly end up in my bad-boys list, moving them from two hour to permanent '3 But for now, this is working well enough
  33. 33. The cost of spamfi ghtin g Many hours of labor to keep things updated Bandwidth overages Excessive CPU load averages . And this is just for a single domain! (Think of the cost at a large ISP. ) False positives also cost me real business
  34. 34. My dream 0 Make junk email illegal (like junk fax/ cell) 0 Culpability for Operating System Producers (read: “microsoft”) for failure to adequately build/ test security, based on actual damages 0 Include “chain of responsibility”, even if junk email comes from outside the US
  35. 35. More resources 0 Magazine articles I've written (Google for “site: stonehenge. com spam”) 0 Documentation for Postfix, AmavisD, SpamAssassin, OpenBSD’s pf 0 Hire me to come give you the longer detailed talk and/ or set up your systems (www. stonehenge. com for contact info)

×