Sonoma State’s
Email
SSU CS Colloquium, October 8, 2009
Eric Eisenhart
<eric.eisenhart@sonoma.edu>
http://eric.eisenhart.name/presentations/

Disclaimers

Disclaimers
•
My opinions

Disclaimers
•My opinions
•No Credit

Disclaimers
•My opinions
•No Credit
•Network == Air

Disclaimers
•My opinions
•No Credit
•Network == Air
•Life

Too Simple

Too Simple
A
Mail Server
#1

Too Simple
A B
Mail Server Mail Server
#1 #2

Big Picture
Alice Bob
Load Balancer #1
ns authns mail ldaps webmail
ns cluster mail cluster authns cluster
moltar zorak blinky snorky
ns-a ns-b ns-c chalmers clancy cletus krusty milhouse authns-a authns-b authns-c
NetApp Cluster
mojo
ns3.csu.net flanders
heckle jeckle
heckle-
jeckle-
shelf-1
shelf-1
heckle- jeckle-
shelf-2 shelf-2

Big Picture Chuck
Red Condor #1 Red Condor #2
Alice Bob Red Condor Offsite
Load Balancer #1
ns authns mail ldaps webmail
ns cluster mail cluster authns cluster
moltar zorak blinky snorky
ns-a ns-b ns-c chalmers clancy cletus krusty milhouse authns-a authns-b authns-c
NetApp Cluster
mojo
ns3.csu.net flanders
heckle jeckle
heckle-
jeckle-
shelf-1
shelf-1
heckle- jeckle-
shelf-2 shelf-2

Simpler Picture?

How Email
Works

An Email
Return-path: <root@eisenhae>
Envelope-to: eisenhae@sonoma.edu
Delivery-date: Wed, 30 Sep 2009 09:57:52 -0700
Received: from exim by nelson with spam-scanned (Exim 4.63)
(envelope-from <root@eisenhae>)
id 1Mt2Ut-0007cJ-M2
for eisenhae@sonoma.edu; Wed, 30 Sep 2009 09:57:52 -0700
X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on nelson
X-Spam-Level: *
X-Spam-Status: No, score=1.6 required=5.0
tests=MISSING_DATE,MISSING_HEADERS,
MISSING_MID autolearn=no version=3.2.5
Received: from eisenhae ([1.2.3.4] helo=eisenhae)
by nelson with esmtp (Exim 4.63)
(envelope-from <root@eisenhae>)
id 1Mt2Uf-0007cC-FN
for eric.eisenhart@sonoma.edu; Wed, 30 Sep 2009 09:57:47 -0700
Subject: test message
Message-Id: <E1Mt2Ut-0007cJ-M2@nelson>
From: root@eisenhae
Date: Wed, 30 Sep 2009 09:57:52 -0700
just a test.

SMTP + MSP
(Simple Mail Transport Protocol +
Mail Submission Protocol)

SMTP + MSP
(Simple Mail Transport Protocol +
Mail Submission Protocol)
220 nelson ESMTP Exim Wed, 30 Sep 2009 09:55:40 -0700

SMTP + MSP
(Simple Mail Transport Protocol +
Mail Submission Protocol)
220 nelson ESMTP Exim Wed, 30 Sep 2009 09:55:40 -0700
EHLO eisenhae

SMTP + MSP
(Simple Mail Transport Protocol +
Mail Submission Protocol)
220 nelson ESMTP Exim Wed, 30 Sep 2009 09:55:40 -0700
EHLO eisenhae
250-nelson Hello eisenhae [1.2.3.4]
250-SIZE 20971520
250-PIPELINING
250-STARTTLS
250 HELP

SMTP + MSP
(Simple Mail Transport Protocol +
Mail Submission Protocol)
220 nelson ESMTP Exim Wed, 30 Sep 2009 09:55:40 -0700
EHLO eisenhae
250-nelson Hello eisenhae [1.2.3.4]
250-SIZE 20971520
250-PIPELINING
250-STARTTLS
250 HELP
MAIL FROM:root@eisenhae.sonoma.edu

SMTP + MSP
(Simple Mail Transport Protocol +
Mail Submission Protocol)
220 nelson ESMTP Exim Wed, 30 Sep 2009 09:55:40 -0700
EHLO eisenhae
250-nelson Hello eisenhae [1.2.3.4]
250-SIZE 20971520
250-PIPELINING
250-STARTTLS
250 HELP
MAIL FROM:root@eisenhae.sonoma.edu
250 OK

SMTP + MSP
(Simple Mail Transport Protocol +
Mail Submission Protocol)
220 nelson ESMTP Exim Wed, 30 Sep 2009 09:55:40 -0700
EHLO eisenhae
250-nelson Hello eisenhae [1.2.3.4]
250-SIZE 20971520
250-PIPELINING
250-STARTTLS
250 HELP
MAIL FROM:root@eisenhae.sonoma.edu
250 OK
RCPT TO:eric.eisenhart@sonoma.edu

SMTP + MSP
(Simple Mail Transport Protocol +
Mail Submission Protocol)
220 nelson ESMTP Exim Wed, 30 Sep 2009 09:55:40 -0700
EHLO eisenhae
250-nelson Hello eisenhae [1.2.3.4]
250-SIZE 20971520
250-PIPELINING
250-STARTTLS
250 HELP
MAIL FROM:root@eisenhae.sonoma.edu
250 OK
RCPT TO:eric.eisenhart@sonoma.edu
250 Accepted

SMTP + MSP
(Simple Mail Transport Protocol +
Mail Submission Protocol)
220 nelson ESMTP Exim Wed, 30 Sep 2009 09:55:40 -0700
EHLO eisenhae
250-nelson Hello eisenhae [1.2.3.4]
250-SIZE 20971520
250-PIPELINING
250-STARTTLS
250 HELP
MAIL FROM:root@eisenhae.sonoma.edu
250 OK
RCPT TO:eric.eisenhart@sonoma.edu
250 Accepted
DATA

SMTP + MSP
(Simple Mail Transport Protocol +
Mail Submission Protocol)
220 nelson ESMTP Exim Wed, 30 Sep 2009 09:55:40 -0700
EHLO eisenhae
250-nelson Hello eisenhae [1.2.3.4]
250-SIZE 20971520
250-PIPELINING
250-STARTTLS
250 HELP
MAIL FROM:root@eisenhae.sonoma.edu
250 OK
RCPT TO:eric.eisenhart@sonoma.edu
250 Accepted
DATA
354 Enter message, ending with "." on a line by itself

SMTP + MSP
(Simple Mail Transport Protocol +
Mail Submission Protocol)
220 nelson ESMTP Exim Wed, 30 Sep 2009 09:55:40 -0700
EHLO eisenhae
250-nelson Hello eisenhae [1.2.3.4]
250-SIZE 20971520
250-PIPELINING
250-STARTTLS
250 HELP
MAIL FROM:root@eisenhae.sonoma.edu
250 OK
RCPT TO:eric.eisenhart@sonoma.edu
250 Accepted
DATA
354 Enter message, ending with "." on a line by itself
Subject: test message
just a test.
.

SMTP + MSP
(Simple Mail Transport Protocol +
Mail Submission Protocol)
220 nelson ESMTP Exim Wed, 30 Sep 2009 09:55:40 -0700
EHLO eisenhae
250-nelson Hello eisenhae [1.2.3.4]
250-SIZE 20971520
250-PIPELINING
250-STARTTLS
250 HELP
MAIL FROM:root@eisenhae.sonoma.edu
250 OK
RCPT TO:eric.eisenhart@sonoma.edu
250 Accepted
DATA
354 Enter message, ending with "." on a line by itself
Subject: test message
just a test.
.
250 OK id=1Mt2Uf-0007cC-FN

SMTP + MSP
(Simple Mail Transport Protocol +
Mail Submission Protocol)
220 nelson ESMTP Exim Wed, 30 Sep 2009 09:55:40 -0700
EHLO eisenhae
250-nelson Hello eisenhae [1.2.3.4]
250-SIZE 20971520
250-PIPELINING
250-STARTTLS
250 HELP
MAIL FROM:root@eisenhae.sonoma.edu
250 OK
RCPT TO:eric.eisenhart@sonoma.edu
250 Accepted
DATA
354 Enter message, ending with "." on a line by itself
Subject: test message
just a test.
.
250 OK id=1Mt2Uf-0007cC-FN
QUIT

SMTP + MSP
(Simple Mail Transport Protocol +
Mail Submission Protocol)
220 nelson ESMTP Exim Wed, 30 Sep 2009 09:55:40 -0700
EHLO eisenhae
250-nelson Hello eisenhae [1.2.3.4]
250-SIZE 20971520
250-PIPELINING
250-STARTTLS
250 HELP
MAIL FROM:root@eisenhae.sonoma.edu
250 OK
RCPT TO:eric.eisenhart@sonoma.edu
250 Accepted
DATA
354 Enter message, ending with "." on a line by itself
Subject: test message
just a test.
.
250 OK id=1Mt2Uf-0007cC-FN
QUIT
221 nelson.sonoma.edu closing connection

POP3
(Post Office Protocol)

POP3
(Post Office Protocol)
+OK POP3 server ready <1896.697170952@dbc.mtview.ca.us>

POP3
(Post Office Protocol)
+OK POP3 server ready <1896.697170952@dbc.mtview.ca.us>
USER mrose

POP3
(Post Office Protocol)
+OK POP3 server ready <1896.697170952@dbc.mtview.ca.us>
USER mrose
+OK User accepted

POP3
(Post Office Protocol)
+OK POP3 server ready <1896.697170952@dbc.mtview.ca.us>
USER mrose
+OK User accepted
PASS mrosepass

POP3
(Post Office Protocol)
+OK POP3 server ready <1896.697170952@dbc.mtview.ca.us>
USER mrose
+OK User accepted
PASS mrosepass
+OK Pass accepted

POP3
(Post Office Protocol)
+OK POP3 server ready <1896.697170952@dbc.mtview.ca.us>
USER mrose
+OK User accepted
PASS mrosepass
+OK Pass accepted
STAT

POP3
(Post Office Protocol)
+OK POP3 server ready <1896.697170952@dbc.mtview.ca.us>
USER mrose
+OK User accepted
PASS mrosepass
+OK Pass accepted
STAT
+OK 2 320

POP3
(Post Office Protocol)
+OK POP3 server ready <1896.697170952@dbc.mtview.ca.us>
USER mrose
+OK User accepted
PASS mrosepass
+OK Pass accepted
STAT
+OK 2 320
LIST

POP3
(Post Office Protocol)
+OK POP3 server ready <1896.697170952@dbc.mtview.ca.us>
USER mrose
+OK User accepted
PASS mrosepass
+OK Pass accepted
STAT
+OK 2 320
LIST
+OK 2 messages (320 octets)

POP3
(Post Office Protocol)
+OK POP3 server ready <1896.697170952@dbc.mtview.ca.us>
USER mrose
+OK User accepted
PASS mrosepass
+OK Pass accepted
STAT
+OK 2 320
LIST
+OK 2 messages (320 octets)
1 120

POP3
(Post Office Protocol)
+OK POP3 server ready <1896.697170952@dbc.mtview.ca.us>
USER mrose
+OK User accepted
PASS mrosepass
+OK Pass accepted
STAT
+OK 2 320
LIST
+OK 2 messages (320 octets)
1 120
2 200

POP3
(Post Office Protocol)
+OK POP3 server ready <1896.697170952@dbc.mtview.ca.us>
USER mrose
+OK User accepted
PASS mrosepass
+OK Pass accepted
STAT
+OK 2 320
LIST
+OK 2 messages (320 octets)
1 120
2 200
.

POP3
(Post Office Protocol)
+OK POP3 server ready <1896.697170952@dbc.mtview.ca.us>
USER mrose
+OK User accepted
PASS mrosepass
+OK Pass accepted
STAT
+OK 2 320
LIST
+OK 2 messages (320 octets)
1 120
2 200
.
RETR 1

POP3
(Post Office Protocol)
+OK POP3 server ready <1896.697170952@dbc.mtview.ca.us>
USER mrose
+OK User accepted
PASS mrosepass
+OK Pass accepted
STAT
+OK 2 320
LIST
+OK 2 messages (320 octets)
1 120
2 200
.
RETR 1
+OK 120 octets

POP3
(Post Office Protocol)
+OK POP3 server ready <1896.697170952@dbc.mtview.ca.us>
USER mrose
+OK User accepted
PASS mrosepass
+OK Pass accepted
STAT
+OK 2 320
LIST
+OK 2 messages (320 octets)
1 120
2 200
.
RETR 1
+OK 120 octets
<the POP3 server sends message 1>

POP3
(Post Office Protocol)
+OK POP3 server ready <1896.697170952@dbc.mtview.ca.us>
USER mrose
+OK User accepted
PASS mrosepass
+OK Pass accepted
STAT
+OK 2 320
LIST
+OK 2 messages (320 octets)
1 120
2 200
.
RETR 1
+OK 120 octets
<the POP3 server sends message 1>
.

POP3
(Post Office Protocol)
+OK POP3 server ready <1896.697170952@dbc.mtview.ca.us>
USER mrose
+OK User accepted
PASS mrosepass
+OK Pass accepted
STAT
+OK 2 320
LIST
+OK 2 messages (320 octets)
1 120
2 200
.
RETR 1
+OK 120 octets
<the POP3 server sends message 1>
.
DELE 1

POP3
(Post Office Protocol)
+OK POP3 server ready <1896.697170952@dbc.mtview.ca.us>
USER mrose
+OK User accepted
PASS mrosepass
+OK Pass accepted
STAT
+OK 2 320
LIST
+OK 2 messages (320 octets)
1 120
2 200
.
RETR 1
+OK 120 octets
<the POP3 server sends message 1>
.
DELE 1
+OK message 1 deleted

POP3
(Post Office Protocol)
+OK POP3 server ready <1896.697170952@dbc.mtview.ca.us>
USER mrose
+OK User accepted
PASS mrosepass
+OK Pass accepted
STAT
+OK 2 320
LIST
+OK 2 messages (320 octets)
1 120
2 200
.
RETR 1
+OK 120 octets
<the POP3 server sends message 1>
.
DELE 1
+OK message 1 deleted
QUIT

POP3
(Post Office Protocol)
+OK POP3 server ready <1896.697170952@dbc.mtview.ca.us>
USER mrose
+OK User accepted
PASS mrosepass
+OK Pass accepted
STAT
+OK 2 320
LIST
+OK 2 messages (320 octets)
1 120
2 200
.
RETR 1
+OK 120 octets
<the POP3 server sends message 1>
.
DELE 1
+OK message 1 deleted
QUIT
+OK dewey POP3 server signing off (maildrop empty)

IMAP
(Internet Message Access Protocol)

IMAP
(Internet Message Access Protocol)
* OK [CAPABILITY IMAP4rev1 UIDPLUS CHILDREN NAMESPACE THREAD=ORDEREDSUBJECT
THREAD=REFERENCES SORT QUOTA IDLE AUTH=PLAIN] Courier-IMAP ready. Copyright
1998-2008 Double Precision, Inc. See COPYING for distribution information.

IMAP
(Internet Message Access Protocol)
* OK [CAPABILITY IMAP4rev1 UIDPLUS CHILDREN NAMESPACE THREAD=ORDEREDSUBJECT
THREAD=REFERENCES SORT QUOTA IDLE AUTH=PLAIN] Courier-IMAP ready. Copyright
1998-2008 Double Precision, Inc. See COPYING for distribution information.
1 AUTHENTICATE PLAIN abcdefghiBASE64jklmnop=

IMAP
(Internet Message Access Protocol)
* OK [CAPABILITY IMAP4rev1 UIDPLUS CHILDREN NAMESPACE THREAD=ORDEREDSUBJECT
THREAD=REFERENCES SORT QUOTA IDLE AUTH=PLAIN] Courier-IMAP ready. Copyright
1998-2008 Double Precision, Inc. See COPYING for distribution information.
1 AUTHENTICATE PLAIN abcdefghiBASE64jklmnop=
1 OK LOGIN Ok.

IMAP
(Internet Message Access Protocol)
* OK [CAPABILITY IMAP4rev1 UIDPLUS CHILDREN NAMESPACE THREAD=ORDEREDSUBJECT
THREAD=REFERENCES SORT QUOTA IDLE AUTH=PLAIN] Courier-IMAP ready. Copyright
1998-2008 Double Precision, Inc. See COPYING for distribution information.
1 AUTHENTICATE PLAIN abcdefghiBASE64jklmnop=
1 OK LOGIN Ok.
2 SELECT INBOX

IMAP
(Internet Message Access Protocol)
* OK [CAPABILITY IMAP4rev1 UIDPLUS CHILDREN NAMESPACE THREAD=ORDEREDSUBJECT
THREAD=REFERENCES SORT QUOTA IDLE AUTH=PLAIN] Courier-IMAP ready. Copyright
1998-2008 Double Precision, Inc. See COPYING for distribution information.
1 AUTHENTICATE PLAIN abcdefghiBASE64jklmnop=
1 OK LOGIN Ok.
2 SELECT INBOX
* FLAGS (Draft Answered Flagged Deleted Seen Recent)
* OK [PERMANENTFLAGS (Draft Answered Flagged Deleted Seen)] Limited
* 751 EXISTS
* 1 RECENT
* OK [UIDVALIDITY 1201564895] Ok
* OK [MYRIGHTS "acdilrsw"] ACL
2 OK [READ-WRITE] Ok

IMAP
(Internet Message Access Protocol)
* OK [CAPABILITY IMAP4rev1 UIDPLUS CHILDREN NAMESPACE THREAD=ORDEREDSUBJECT
THREAD=REFERENCES SORT QUOTA IDLE AUTH=PLAIN] Courier-IMAP ready. Copyright
1998-2008 Double Precision, Inc. See COPYING for distribution information.
1 AUTHENTICATE PLAIN abcdefghiBASE64jklmnop=
1 OK LOGIN Ok.
2 SELECT INBOX
* FLAGS (Draft Answered Flagged Deleted Seen Recent)
* OK [PERMANENTFLAGS (Draft Answered Flagged Deleted Seen)] Limited
* 751 EXISTS
* 1 RECENT
* OK [UIDVALIDITY 1201564895] Ok
* OK [MYRIGHTS "acdilrsw"] ACL
2 OK [READ-WRITE] Ok
3 FETCH 1 BODY[]

IMAP
(Internet Message Access Protocol)
* OK [CAPABILITY IMAP4rev1 UIDPLUS CHILDREN NAMESPACE THREAD=ORDEREDSUBJECT
THREAD=REFERENCES SORT QUOTA IDLE AUTH=PLAIN] Courier-IMAP ready. Copyright
1998-2008 Double Precision, Inc. See COPYING for distribution information.
1 AUTHENTICATE PLAIN abcdefghiBASE64jklmnop=
1 OK LOGIN Ok.
2 SELECT INBOX
* FLAGS (Draft Answered Flagged Deleted Seen Recent)
* OK [PERMANENTFLAGS (Draft Answered Flagged Deleted Seen)] Limited
* 751 EXISTS
* 1 RECENT
* OK [UIDVALIDITY 1201564895] Ok
* OK [MYRIGHTS "acdilrsw"] ACL
2 OK [READ-WRITE] Ok
3 FETCH 1 BODY[]
* 1 FETCH (BODY[] {2390}
<server sends full message with headers>
3 OK FETCH completed.

IMAP
(Internet Message Access Protocol)
* OK [CAPABILITY IMAP4rev1 UIDPLUS CHILDREN NAMESPACE THREAD=ORDEREDSUBJECT
THREAD=REFERENCES SORT QUOTA IDLE AUTH=PLAIN] Courier-IMAP ready. Copyright
1998-2008 Double Precision, Inc. See COPYING for distribution information.
1 AUTHENTICATE PLAIN abcdefghiBASE64jklmnop=
1 OK LOGIN Ok.
2 SELECT INBOX
* FLAGS (Draft Answered Flagged Deleted Seen Recent)
* OK [PERMANENTFLAGS (Draft Answered Flagged Deleted Seen)] Limited
* 751 EXISTS
* 1 RECENT
* OK [UIDVALIDITY 1201564895] Ok
* OK [MYRIGHTS "acdilrsw"] ACL
2 OK [READ-WRITE] Ok
3 FETCH 1 BODY[]
* 1 FETCH (BODY[] {2390}
<server sends full message with headers>
3 OK FETCH completed.
4 LOGOUT

IMAP
(Internet Message Access Protocol)
* OK [CAPABILITY IMAP4rev1 UIDPLUS CHILDREN NAMESPACE THREAD=ORDEREDSUBJECT
THREAD=REFERENCES SORT QUOTA IDLE AUTH=PLAIN] Courier-IMAP ready. Copyright
1998-2008 Double Precision, Inc. See COPYING for distribution information.
1 AUTHENTICATE PLAIN abcdefghiBASE64jklmnop=
1 OK LOGIN Ok.
2 SELECT INBOX
* FLAGS (Draft Answered Flagged Deleted Seen Recent)
* OK [PERMANENTFLAGS (Draft Answered Flagged Deleted Seen)] Limited
* 751 EXISTS
* 1 RECENT
* OK [UIDVALIDITY 1201564895] Ok
* OK [MYRIGHTS "acdilrsw"] ACL
2 OK [READ-WRITE] Ok
3 FETCH 1 BODY[]
* 1 FETCH (BODY[] {2390}
<server sends full message with headers>
3 OK FETCH completed.
4 LOGOUT
* BYE Courier-IMAP server shutting down
4 OK LOGOUT completed

LDAP
(Lightweight Directory Access Protocol)
dn: uid=eisenhae,ou=people,o=sonoma,o=edu
uid: eisenhae
ssuMailHomeDirectory: /nfs/jeckle/maildirs/92/eisenhae
ssuMailLocalAddress: eisenhae@sonoma.edu
ssuMailLocalAddress: eric.eisenhart@sonoma.edu
ssuMailRoutingAddress: eisenhae@smtp-vip.sonoma.edu
ssuMailSpamAssassinPrefs: required_hits 5
ssuMailSpamAssassinPrefs: whitelist_from *@example.com
ssuMailDiskQuota: 1024M

Maildir (++)
/nfs/jeckle/maildirs/92/eisenhae/
courierimapsubscribed
courierimapuiddb
tmp/
new/
new/1254971281.H995082P22240.cletus.sonoma.edu,S=3593
cur/
cur/1251296264.H25153P29370.krusty.sonoma.edu,S=3045:2,RS
.GrayMai/cur/
.GrayMail/new/
.GrayMail/tmp/
.Lists/
.Lists.puppet/

The
aka: requirements

Email ==
“Sonoma State University
will employ these email
accounts for official
University
communications to
students effective
immediately.”
http://www.sonoma.edu/it/get_connected/
provost_letter.shtml

Reliance

Reliance
Assumed

Reliance
Assumed
Addicted

Growth

Population

1500
1125
GB
750
375
0
2004 2005 2006 2007 2008 2009
~300G 700G 1.35T
Storage Usage

SPA

SPA

Traffic
2006-08 to now-

Peak Traffic
2006-08 to now-

Moore’s Law:
Friend or Foe?

The
aka: what we did

Limits

Limits
•
Money

Limits
•Money
•Time / Staff

Goals

Goals
•
Reliable

Goals
•Reliable
•Minimal Update Costs

Goals
•Reliable
•Minimal Update Costs
•Concurrency

Goals
•Reliable
•Minimal Update Costs
•Concurrency
•Asynchronous

What We Run

What we Run
Hardware

What we Run
Hardware
• MAIL: 5 + test. 8GB, 2 dual-core 2.4GHz AMD

What we Run
Hardware
• MAIL: 5 + test. 8GB, 2 dual-core 2.4GHz AMD
• LDAP: 2 + master + standby. 2GB, 2 Intel
3GHz

What we Run
Hardware
• MAIL: 5 + test. 8GB, 2 dual-core 2.4GHz AMD
• LDAP: 2 + master + standby. 2GB, 2 Intel
3GHz
• webmail: 2 VMs. 2GB

What we Run
Hardware
• MAIL: 5 + test. 8GB, 2 dual-core 2.4GHz AMD
• LDAP: 2 + master + standby. 2GB, 2 Intel
3GHz
• webmail: 2 VMs. 2GB
• lists: 1 VM. 1GB

What we Run
Hardware
• MAIL: 5 + test. 8GB, 2 dual-core 2.4GHz AMD
• LDAP: 2 + master + standby. 2GB, 2 Intel
3GHz
• webmail: 2 VMs. 2GB
• lists: 1 VM. 1GB
• MySQL: 1 VM. 1GB.

What we Run
Hardware
• MAIL: 5 + test. 8GB, 2 dual-core 2.4GHz AMD
• LDAP: 2 + master + standby. 2GB, 2 Intel
3GHz
• webmail: 2 VMs. 2GB
• lists: 1 VM. 1GB
• MySQL: 1 VM. 1GB.
• storage: NetApp FAS 3070c cluster; 30*136GB
15k

Virtual Machines?

What we Run
“MAIL”

What we Run
“MAIL”
•exim

What we Run
“MAIL”
•exim
•clamav

What we Run
“MAIL”
•exim
•clamav
•SpamAssassin

What we Run
“MAIL”
•exim
•clamav
•SpamAssassin
•courier-auth

What we Run
“MAIL”
•exim
•clamav
•SpamAssassin
•courier-auth
•courier-imap

What we Run
Storage

What we Run
Storage
•2 * NetApp FAS 3070s in
active-active cluster

What we Run
Storage
•2 * NetApp FAS 3070s in
active-active cluster
•10 shelves of 15 disks

What we Run
Storage
•2 * NetApp FAS 3070s in
active-active cluster
•10 shelves of 15 disks
•EMC NAS (going, going, gone)

What we Run
“webmail”

What we Run
“webmail”
•apache httpd

What we Run
“webmail”
•apache httpd
•php

What we Run
“webmail”
•apache httpd
•php
•squirrelmail + plugins

What we Run
“webmail”
•apache httpd
•php
•squirrelmail + plugins
•up-imapproxy

What we Run
“ldaps”

What we Run
“ldaps”
•apache httpd

What we Run
“ldaps”
•apache httpd
•OpenLDAP (slapd)

What we Run
“ldaps”
•apache httpd
•OpenLDAP (slapd)
•Lots of custom code

What we Run
“flanders”

What we Run
“flanders”
•mysqld

What we Run
“lists”

What we Run
“lists”
•apache httpd

What we Run
“lists”
•apache httpd
•mailman

What we Run
Secret Sauces

What we Run
Secret Sauces
•Tsunami

What we Run
Secret Sauces
•Tsunami
•puppet

What we Run
Secret Sauces
•Tsunami
•puppet
•Registry

SPAM

SPAM
•
SpamAssassin

SPAM
•SpamAssassin
•Red Condor

Changes (so far)
• storage backend (wiggum; otto;
bruno; heckle+jeckle)
• Directory layout (f/foo -> ac/foo)
• Red Condor
• IMAP + SMTP + SPAM + clamav
(same servers now)
• SPAM prefs from MySQL to LDAP
• Virtualization

The Past

The Past
• My First: one box

The Past
• My First: one box
• VAX; cluster

The Past
• My First: one box
• VAX; cluster
• 2003: VAX + student server

The Past
• My First: one box
• VAX; cluster
• 2003: VAX + student server
• Later 2003: wiggum -> otto

The Past
• My First: one box
• VAX; cluster
• 2003: VAX + student server
• Later 2003: wiggum -> otto
• Summer 2005: performance?

Failures
aka: what went wrong

Spring 2006

Fall 2007-Spring
2008

Problems

Problems
•
Cache

Problems
•Cache
•SPAM accuracy

Problems
•Cache
•SPAM accuracy
•Local SPAM

Problems
•Cache
•SPAM accuracy
•Local SPAM
•Quotas

Alternatives /
Future
• Smarter Load Balancers
• IMAP server with local caching
• Shared block storage (mini
clusters?)
• Exchange
• Outsourced (Google)

Building your own?
• Plan for Growth
• Understand how to scale everything
O(1) > O(log(n)) > O(n)

Questions!